
Calhoun 

imU'uiioiu! A Kim of die Nav.il Po<ic^rdduacc School 


Calhoun: The NPS Institutional Archive 
DSpace Repository 



Theses and Dissertations 


1. Thesis and Dissertation Collection, all items 


1995-03 

Streamlining of Army Acquisition Category 
(ACAT) IV programs 

Doucette, Edward John 

Monterey, California. Naval Postgraduate School 


http://hdl.handle.net/10945/31538 


This publication is a work of the U.S. Government as defined in Title 17, United 
States Code, Section 101. Copyright protection is not available for this work in the 
United States. 

Downloaded from NPS Archive: Calhoun 



DUDLEY 

KNOX 

LIBRARY 


http ://w w w. nps.edu/lEbrary 


Calhoun is the Naval Postgraduate Sc hoof's public access digital repository for 
research mate hate and institutional publications created by tire NPS community, 
Calhoun is named for Professor of Mathematics Guy K. Calhoun, NPS's first 
appointed — and published — scholar^ author. 

Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 University Cirde 
Monterey, California USA 93943 



NAVAL POSTGRADUATE SCHOOL 
MONTEREY, CALIFORNIA 



STREAMLINING OF 
ARMY 

ACQUISITION CATEGORY (ACAT) IV 
PROGRAMS 

by 

Edward John Doucette 
March, 1995 

Principal Advisor: George Prueitt 

Approved for public release; distribution is unlimited. 


19950821 049 







REPORT DOCUMENTATION PAGE 


Form Approved OMB No. 0704-0188 


Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching 
existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this 
burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, 
Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management 
and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503. 

T. AGENCY USE ONLY (Leave blank) I 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED 

March, 1995 Master’s Thesis 

4. TITLE AND SUBTITLE STREAMLINING OF ARMY 5. FUNDING NUMBERS 

ACQUISITION CATEGORY (ACAT) IV PROGRAMS _ 

6. AUT HOR(S) Edward John Doucette _ 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING 

Naval Postgraduate School ORGANIZATION 

Monterey CA 93943-5000 _ REPORT NUMBER _ 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING/MONITORING 

U.S. Army Natick RD&E Center, Natick, MA 01760 _ AGENCY REPORT NUMBER 

11. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect 
the official policy or position of the Department of Defense or the U.S. Government. 

12a. DISTRIBUTION/AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE 

Approved for public release; distribution is unlimited. _ 

13. ABSTRACT (maximum 200 words) 

The 1986 Packard Commission review of Defense management recommended a consolidation of the 
Services’ materiel acquisition regulations under an umbrella regulation, DoD 5000. Adopted in 1991, this 
regulation created four levels of management oversight. Acquisition Categories (ACAT) I - IV, and it 
encouraged streamlining of the process for the smaller ACAT III and IV programs. 

This study examines the streamlining of DoD 5000 procedures across the smallest U.S. Army 
programs (ACAT IV). It investigates differences among Commands and streamlining techniques that are 
most prevalent and useful. The basis of the study consisted of literature searches, interviews, and a 
questionnaire, with questions that required numeric and essay responses, that was distributed Army-wide to 
all ACAT IV managers, and synthesis of this collection of information. A total of 19 managers responded. 

The study concluded that there are no significant differences among the U.S. Army Commands 
concerning the application of acquisition streamlining techniques. In addition, it identified which techniques 
were most prevalent, which saved the most program time and funding, and the reasons for differences 
among Commands and between small and large programs. The study recommends that the Army establish 
an ACAT IV baseline development management process that incorporates the most prevalent and useful 
streamlining techniques. 

14. SUBJECT TERMS acquisition, Army, development, streamlining, tailoring 15. NUMBER OF 

PAGES 133 

16. PRICE CODE 

17. SECURITY CLASSIFI- I 18. SECURITY CLASSIFI- 19. SECURITY CLASSIFI- 20. LIMITATION OF 
CATION OF REPORT CATION OF THIS PAGE CATION OF ABSTRACT ABSTRACT 

Unclassified Unclassified Unclassified UL 

NSN 7540-01-280-5500 "" Standard Form 298 (Rev. 2-89) 

Prescribed by ANSI Std. 239-18298-102 


1 


DUG imFECTED g, 






















Approved for public release; distribution is unlimited. 

STREAMLINING OF ARMY 
ACQUISITION CATEGORY (ACAT) IV 
PROGRAMS 

by 

Edward John Doucette 
Department of the Army 
B.S., University of Massachusetts, 1980 

Submitted in partial fulfillment 
of the requirements for the degree of 

MASTER OF SCIENCE IN MANAGEMENT 

from the 



Author: 


Approved by: 


NAVAL POSTGRADUATE SCHOOL 
March 1995 



George Prueitt, Principal Advisor 



Mark Stone, Associate Advisor 



ill 









ABSTRACT 


The 1986 President’s Blue Ribbon Commission on Defense Management (Packard 
Commission) recommended a consolidation of the Services’ materiel acquisition 
regulations under an umbrella regulation, DoD 5000. Adopted in 1991, this regulation 
created four levels of management oversight, Acquisition Categories (ACAT) I - IV, and 
it encouraged streamlining or tailoring of the process for the smaller ACAT III and IV 
programs. 

This study examines the streamlining of the DoD 5000 procedures across the 
smallest U.S. Army programs (ACAT IV). It investigates differences among Commands 
and which streamlining techniques are most prevalent and useful. The basis of the study 
consisted of literature searches, personal and telephonic interviews, and a questionnaire, 
with questions that required numeric and essay responses, that was distributed Army-wide 
to all ACAT IV managers, and synthesis of this collection of information. A total of 19 
managers responded. 

The study concluded that there are no significant differences among the U.S. Army 
Commands concerning the application of acquisition streamlining techniques. In addition, 
it identified which techniques were most prevalent, which saved the most program time 
and funding, and the reasons for differences among Commands and between small and 
large programs. The study recommends that the Army establish an ACAT IV baseline 
development management process that incorporates the most prevalent and useful 
streamlining techniques. 
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EXECUTIVE SUMMARY 


In the late 1980’s, the President’s Blue Ribbon Commission on Defense 
Management (Packard Commission) in A Quest for Excellence, Final Report to the 
President recommended, and the Department of Defense (DoD) Defense Management 
Review instituted, a consolidation of the Services’ materiel acquisition regulations under 
an umbrella regulation, DoD 5000. This regulation was adopted in 1991 and it changed 
many aspects of how the Services manage and conduct material development programs. 
The system adopted by DoD created four levels of management oversight, Acquisition 
Categories (ACAT) I - IV. 

The new DoD 5000 series concentrates on the management oversight required 
for large ACAT I programs. The regulations outline the processes required of AC AT I 
managers, and encourages tailoring, or streamlining of the process by ACAT II, III, and 
IV managers, while not allowing for across-the-board Service-level policies. 

The purpose of this research was to analyze the streamlining of DoD 5000 
procedures and processes across all of the smallest U.S. Army programs (ACAT IV); 
to investigate differences among various U.S. Army Commands and determine which 
streamlining techniques were most prevalent, useful, and the circumstances that cause 
them to be successful. Individual managers need to streamline to effectively manage 
program risk, but they lack key data necessary for conducting proper risk assessments. 
This study sought to provide this missing risk assessment data. Specific purposes were 
to determine whether Commands need assistance in gaining knowledge concerning all of 
the streamlining techniques available (being used by others) and to provide the ACAT 
IV managers with tools to assist them in their streamlining decision-making. 

Primary Research Question: What acquisition streamlining techniques are utilized by 
U.S. Army ACAT IV managers? 
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Subsidiary Research Questions: 

(1) How do U.S. Army Commands streamline ACATIV programs differently? - 
Comparing the techniques utilized Command-by-Command . 

(2) How do the various streamlining techniques compare with respect to program 

cost savings? - Comparing technique-to-technique. 

(3) How do the various streamlining techniques compare with respect to program 

schedule reduction? - Comparing technique-to-technique. 

The methodology utilized for this effort was a management problem analysis using 
both quantitative and qualitative techniques. The qualitative techniques involved the use 
of interviews to determine the opinions of the AMC acquisition policy Subject Matter 
Experts concerning the validity and breadth of the techniques to be investigated. The 
information from this process was assimilated into a broad-based questionnaire that was 
distributed Army-wide to all ACAT IV development managers, with questions that 
required numeric and essay responses. The numeric responses were quantitatively 
analyzed, and the essay responses were used to find elaboration/causal relationships for 
specific responses and/or trends observed in the data. 

In support of this thesis, 39 questionnaires were distributed to the ACAT IV 
development managers at various U.S. Army AMC Commands. Two Commands, 
CECOM and MICOM, were eliminated from the survey as is was determined that all of 
their development programs were at the ACAT III level or higher. A total of 19 
responses were obtained, with some Commands providing limited responses. The data 
that was received was deemed to be valid, and the analysis was conducted on a 
predominately qualitative basis. 

From the study, it was concluded that cross-fertilization of streamlining concepts 
exists among the U.S. Army Commands. The information is available for all Commands 
to use. In addition, relative usage levels for the various techniques is consistent among 
Natick and the other U.S. Army Commands. For those techniques where the usage 
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differed the most, the reasons were disagreement with the value or risk of the technique, 
or specific commodity/Command differences. 

A number of streamlining techniques are used routinely by AC AT IV programs. 
The risk levels associated with the combining of Milestones and test phases are 
considered acceptable for programs at this lower level of complexity. Although 
quantifiable and useful ranges for the levels of anticipated savings for the techniques 
could not be projected from the research, the following can be concluded as the key 
techniques utilized by most AC AT IV managers. 

1. Combining of Milestones 0 and I, and/or I and II. 

2. Combining of test phases, both in Dem/Val and EMD, for Natick managers. 

3. Concurrency of test phases, both in Dem/Val and EMD, for non-Natick 

managers. 

4. Combining development deficiency correction validation testing with First 

Article Testing. 

Small and large ACAT IV program managers are in close agreement concerning 
the relative cost savings associated with each of the streamlining techniques in the study. 
The two areas where large programs deviated from the trend are: (1) combining EMD 
DT and IOT&E, and (2) using a single contract for Dem/Val and EMD, both of which 
may be due to the increased complexity of larger programs. 

It is recommended that the results of this study be broadly distributed to ACAT 
IV managers. Even though the results provide no quantitative assistance for ACAT IV 
managers concerning expected savings levels for the various streamlining techniques 
investigated, the study does provide valuable information that should assist managers in 
their streamlining decision-making. 

It is further recommended that the U.S. Army revise the baseline development 
process for ACAT IV programs, to one that incorporates the above key streamlining 
techniques as standard procedures from which requirements are either increased, or 
decreased. In place of the current system, where the manager must begin with the full 
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AC AT I/II process and straggle with the acquisition team members to cut out elements, 
this recommendation would put acquisition team members in a position of justifying the 
need for adding requirements versus the manager having to justifying their exclusion. 



I. INTRODUCTION 


In the late 1980’s, a Department of Defense (DoD) review of acquisition 
management practices resulted in the consolidation of the Services’ materiel acquisition 
regulations under an umbrella regulation, DoD 5000. This regulation was adopted in 
1991 and it changed many aspects of how the Services manage and conduct material 
development programs. The system adopted by DoD created four levels of management 
oversight, Acquisition Categories (ACAT) I - IV. Application of these categories is based 
on program size and/or the special interests of the Office of the Secretary of Defense and 
the Service’ headquarters. The largest and most significant programs are ACAT I and, 
conversely, the smallest and least significant are ACAT IV. [Ref. 1] 

One aspect of the new DoD 5000 series that did not exist in the old regulations 
is that the Services are now prohibited from regulatory supplementation. This was 
instituted to specifically eliminate the possibility of the Services reverting to Service- 
specific policy. Unfortunately, it also creates an inability for a Service to implement an 
across-the-board policy for streamlining the procedures and processes outlined in DoD 
5000 without OSD approval. DoD 5000 does, however, encourage streamlining or 
tailoring the process on a project-by-project basis. [Ref. 2] 

A. PROBLEM DEFINITION 

The purpose of this research is to analyze the streamlining of DoD 5000 
procedures and processes across all U.S. Army ACAT IV programs as well as to 
investigate differences among Commands and determine which streamlining techniques 
are most prevalent, useful, and the circumstances that cause them to be successful. From 
the researcher’s twelve years of experience in ACAT IV development management and 
discussions with colleagues within the U.S. Army, streamlining of ACAT IV programs 
has been taking place, but on an ad-hoc, trial-and-error basis. Comparison, or cross- 
fertilization, of information among individual managers and Commands (concerning 
which streamlining techniques have, or have not worked) has only taken place through 
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informal channels. It can be expected that within each Command, the information has 
been disseminated fairly well, since each Command has a support organization on 
acquisition policy that is involved in each project’s planning. [Ref. 3] However, the ability 
to pass ideas between Commands may be incomplete, since there is not a formal means 
to channel this information. Therefore, this study investigates the various streamlining 
techniques utilized by ACAT IV managers among the various Commands and seek to 
define the causal relationships for their use. Specific purposes are to determine whether 
Commands need assistance in gaining knowledge concerning the streamlining techniques 
available and to provide the ACAT IV managers with tools to assist them in their 
streamlining decision-making. This study draws conclusions regarding which techniques 
are most useful, which need to be advertised to other Commands, and the key factors 
that influence the streamlining decision-making process. 

1. Research Questions 

Primary Research Question: What acquisition streamlining techniques are 
utilized by U.S. Army ACAT IV managers? 

Subsidiary Research Questions: 

(1) How do U.S. Army Commands streamline ACAT IV programs differently? 
(Comparing the techniques utilized Command-by-Command). 

(2) How do the various streamlining techniques compare with respect to program 
cost savings? (Comparing technique-to-technique). 

(3) How do the various streamlining techniques compare with respect to program 
schedule reduction? (Comparing technique-to-technique). 

2. Expected Benefit 

This research is intended to determine if tools exist that ACAT IV managers can 
use to improve their acquisition streamlining decisions, and, if so, provide these 
managers with alternative streamlining techniques not previously considered by their 
Commands. The results are also intended to assist policy makers and support commands 
when they consider the adoption of streamlining policies for ACAT IV programs, or their 
application to new program starts. 
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This research effort investigates one aspect of the management of small programs, 
the streamlining techniques that the experts, the current AC AT IV managers, have 
deemed beneficial to their programs. This research is unique in that it focuses on the 
smallest acquisition programs. As noted by the lack of related research discovered in the 
literature search for this research project (see the literature search results below), efforts 
at acquisition reform and streamlining have been almost exclusively concentrated on the 
high-visibility, high-cost weapon system acquisitions (the AC AT I and IIs). Thus, the 
smaller programs have not been considered in depth, and they have been required to use 
the DoD 5000 policies and procedures tailored for the larger programs. This effort is 
sponsored by the U.S. Army Natick Research, Development, and Engineering Center, 
the organization containing the majority of the U.S. Army’s AC AT IV development 
programs. 

3. Boundaries 

This study investigates acquisition streamlining techniques within the context of 
tailoring acquisition procedures and documentation, per DoD 5000 direction, for the 
development of new U.S. Army ACAT IV systems. The research includes analysis of 
all available on-going and recently completed U.S. Army ACAT IV development 
programs, except as noted in the limitations and constraints below. This is not an 
evaluation of the use of commercial versus military specifications and standards, and it 
excludes non-developmental and classified ACAT IV programs, as well as non-regulatory 
program acceleration techniques such as improved business management processes, Total 
Quality Management, program management software tools, etc. 

4. Limitations and Constraints 

Programs that are in production are excluded, because of the difficulty in 
accessing past managers (who have been reassigned) and the time and funding constraints 
of this research effort. However, this should not be a significant limitation since the 
ACAT system has only existed for three years, and these programs largely would have 
been developed under the old system. Also, the amount of funding and time available did 
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not allow for travel to conduct face-to-face interviews with each of the ACAT IV 
managers. 

As will be discussed later in this paper, the main support instrument for this 
research, a written questionnaire, had a low response rate. Investigation of the lack of 
response found that: there are not as many ACAT IV development managers as 
anticipated, some managers are not as knowledgeable as assumed, and some managers 
did not have time available to complete the questionnaire. Because of this limitation, the 
analysis is predominately qualitative, instead of quantitative, in nature. 

5. Assumptions 

It is assumed, from the researcher’s experience at the U.S. Army Natick 
Research, Development, and Engineering Center, that streamlining is a concern of all 
U.S. Army ACAT IV managers. It is also assumed that the ACAT IV managers are the 
most knowledgeable as to which streamlining techniques are most effective, because it 
is their mission to manage these decisions, and they should have an established 
knowledge base from their experiences and discussions with technical support staffs. 

Additional assumptions are as follows: generalized relationships exist among 
ACAT IV programs (with respect to streamlining initiatives) that can be analyzed within 
the constraints of this study; as all ACAT IV programs follow the same general process 
(and are relatively low risk) qualitative comparisons of ACAT IV processes and 
streamlining techniques should be possible; managers aren’t able to exactly quantify the 
impacts of individual streamlining initiatives; and a sufficient quantity of ACAT IV 
managers exist, and a sufficient number of the managers are willing to participate, to 
provide a meaningful basis for analysis. 

B. METHODOLOGY 

The approach utilized for this effort was a management problem analysis through 
a survey of all U.S. Army ACAT IV project managers/officers for their expert opinions 
concerning streamlining techniques. They were asked if they have, or would utilize 
specific streamlining initiatives, and rate/comment on the cost and schedule savings 
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associated with each. Also, comments were requested concerning the key factors that led 
to the above ratings. Results were then evaluated comparatively, both by Command and 
by technique. 

Because the population of AC AT IV managers is geographically dispersed, the 
data collection methods used were telephone interviews and written questionnaires. The 
broad, investigative nature of the questions being addressed in this initial study of AC AT 
IV management made this methodology acceptable. In addition, the effects of this 
constraint were mitigated through careful construction of the questionnaire materials. For 
instance, the telephone interviews established a mutually exclusive and exhaustive set of 
streamlining techniques for use in the written questionnaires (allowing for closed-ended 
questions), and the size of the respondents programs were established (in terms of cost 
and duration) to ensure that the comparative analysis encompassed programs of similar 
size. 

C. LITERATURE REVIEW 

A review of the available literature revealed no directly related past efforts. This 
result is understandable, given the general focus within the DoD on the major systems, 
and the newness of the ACAT designation system. 

A number of previous studies did consider acquisition streamlining, but at a much 
broader level. One of these reviews was the President’s Blue Ribbon Commission of 
1986, discussed in Chapter II, which was the impetus for the new DoD 5000 series. 
These studies were, however, non-analytical, expert panel reviews of the overall process. 
None analyzed streamlining across projects, and none considered non-major, or ACAT 
IV programs. 

One previous study was closely related to the purpose and methodology of this 
research effort. A 1991 study by Mr. Henry Jehan, for the U.S. Army War College, 
examined the impact of the Packard Commission on the U.S. Army ACAT I, II, and III 
managers (the population of U.S. Army development managers specifically not within 
the scope of the current research effort). Mr. Jehan utilized a questionnaire to all 
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available members of the population (the Program and Product Managers), which 
corresponded closely to the methodology planned for this effort. Mr. Jehan’s research 
questions did not, however, investigate the same issues as the current research. His 
research question was stated as follows: 

The central question this study attempts to answer is: how well has 
the Army implementation embodied the recommendations of the Packard 
Commission? Specifically, do the current Army management structures 
conform to the structures recommended by the Commission? And more 
importantly, does the Army acquisition process implement the spirit of the 
Commission’s recommendations?[Ref. 4] 

Mr. Jehan s report provides a useful general outline for conducting the current 
study. It confirms the validity of the methodology and it contains data sources and 
questionnaire lessons learned that can be applied to this research. 

The two previous acquisition streamlining theses at the Naval Postgraduate School 
were completed prior to the advent of the ACAT system. Those studies focused on the 
investigation of streamlining within a specific program/Command and they consisted 
mainly of historical assessments of that organization’s streamlining actions. The first, 
written by Mary Walsh and published in December 1986, utilized reviews of regulations 
and program documents, as well as interviews and resulted in a case study. The purpose 
was to analyze the effectiveness of the then new Army Streamlined Acquisition Program 
(ASAP), the Navy T-45 Trainer development program, and the Army and Navy 
implementation of the since superseded DoDI 5000.43, "Acquisition 
Streamlining".[Ref. 5] The second, written by Michelle McKeever and published in June 
1987, reviewed the acquisition streamlining activities within the Space and Naval Warfare 
Systems Command. It also followed a case study methodology using interviews and 
document reviews. [Ref. 6] 

From this literature review, and the researcher’s experience with past U.S. Army 
management studies of the acquisition process that all concentrated on major weapon 
systems, it is felt that this research is in a new area of investigation. 
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D. SUMMARY OF FINDINGS 


1. Conclusions 

From this study, it can be concluded that cross-fertilization of streamlining 
concepts exists among the U.S. Army Commands. The information is available for all 
Commands to use. In addition, relative usage levels for the various techniques is 
consistent among Natick and the other U.S. Army Commands. For those techniques 
where the usage differed the most, the reasons were disagreement with the value or risk 
of the technique, or specific commodity/Command differences. 

A number of streamlining techniques are used routinely by AC AT IV programs. 
The risk levels associated with the combining of Milestones and test phases are 
considered acceptable for programs at this lower level of complexity. Although 
quantifiable and useful ranges for the levels of anticipated savings for the techniques 
could not be projected from the research, the following can be concluded as the key 
techniques utilized by most AC AT IV managers. 

1. Combining of Milestones 0 and I, and/or I and II. 

2. Combining of test phases, both in Dem/Val and EMD, for Natick managers. 

3. Concurrency of test phases, both in Dem/Val and EMD, for non-Natick 

managers. 

4. Combining development deficiency correction validation testing with First 

Article Testing. 

In addition, ACAT IV programs generally do not require COEA or STAR 
program documents, and the acceleration of independent evaluation and assessment 
reporting for Milestone Decision Reviews is commonly done to accelerate the process 
during the time-sensitive weeks prior to a major decision review. 

Small and large ACAT IV program managers are in close agreement concerning 
the relative cost savings associated with each of the streamlining techniques in the study. 
The two areas where large programs deviated from the trend are: (1) combining EMD 
DT and IOT&E, and (2) using a single contract for Dem/Val and EMD, both of which 
may be due to the increased complexity of larger programs. 
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2. Recommendations Regarding ACAT IV Management 

It is recommended that the results of this study be broadly distributed to ACAT 
IV managers. Even though the results provide no quantitative assistance for ACAT IV 
managers concerning expected savings levels for the various streamlining techniques 
investigated, the study does provide valuable information that should assist managers in 
their streamlining decision-making. 

It is further recommended that the U.S. Army revise the baseline development 
process for ACAT IV programs, to one that incorporates the above key streamlining 
techniques of combining of Milestones I and II concurrent test phases in EMD as 
standard procedures from which requirements are either increased, or decreased. In 
addition, it is recommended that the alternative process exclude the requirements for 
COEA and STAR, except on an exception basis only. In place of the current system, 
where the manager must begin with the full ACAT I/II process and struggle with the 
acquisition team members to cut out elements, this recommendation would put acquisition 
team members in a position of justifying the need for adding requirements versus the 
manager having to justifying their exclusion. 

3. Recommendations for Further Research 

Considering the lack of statistically significant results from this study, it is 
recommended that further study be conducted that concentrates on the most significant 
techniques, investigating more in-depth the establishment of useful statistics, such as 
confidence intervals, for the expected levels of cost and schedule savings. 

Based on the small number of ACAT IV managers in the U.S. Army it is 
recommended that future research in this area consider both ACAT III and ACAT IV 
managers to study the differences between the two groups, and, if they prove to be 
similar, combine the two for statistical information concerning expected savings. 

Since the Marine Corps has many more ACAT III and IV programs than ACAT 
I and II, and they are actively streamlining wherever possible, it is recommended that 
further research be conducted that considers both Army and Marine Corps programs. 


8 




H. BACKGROUND 


A. THE DOD 5000 SERIES 

During the early 1980s, major problems with DoD management of weapon 
systems acquisitions had become issues of concern within the Executive Branch and 
Congress. In addition, the press and the American public were raising concerns of 
perceived fraud, waste, and abuse. Cost overruns on weapon systems, outrageous spare 
parts prices, and alleged fraudulent activities within the Department and it’s contractors 
brought an overwhelming desire for reforms. [Ref. 7] In response to these problems, by 
Executive Order 12526, President Reagan chartered a commission in July 1985 to 
conduct a study of Department of Defense management problems. The President’s Blue 
Ribbon Commission on Defense Management, more popularly known as the "Packard 
Commission" after the chairman of the Commission, former Deputy Secretary of Defense 
Mr. David Packard, took a broad and in-depth look at defense weapon system acquisition 
issues and their root causes. Their charter said that: 

The Commission shall study the issues surrounding defense 
management and organization.... The primary objective of the 
Commission shall be to study defense management policies and 
procedures, including the budget process, the procurement system, 
legislative oversight, and the organizational and operational arrangements, 
both formal and informal, among the Office of the Secretary of Defense, 
the Organization of the Joint Chiefs of Staff, the unified and Specified 
Command system, the Military Departments, and the Congress. In 
particular, the Commission shall: 

1. Review the adequacy of the defense acquisition process, 
including the adequacy of the defense industrial base, current law 
governing Federal and Department of defense procurement activities, 
departmental directives and management procedures, and the execution of 
acquisition responsibilities within the Military Departments. (Emphasis 
added) [Ref. 8] 

The goals of the Commission included strengthening and streamlining the control 
and supervision of the acquisition process. One of the many significant recommendations 
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proposed by the Commission in their June 1986 report was the consolidation and 
simplification of acquisition laws and regulations. [Ref. 9] A follow-on review of the 
regulations governing the Service’s acquisition processes led to a July 1989 Defense 
Management Review (DMR) Decision to consolidate the material acquisition regulations 
of the Services under an umbrella set of regulations. The basis for this effort was to 
eliminate unnecessary Service-unique oversight and bureaucracy, and streamline the 
overall process.[Ref. 10J 

The new DoD Directive 5000.1, "Defense Acquisition"; DoD Instruction 5000.2, 
"Defense Acquisition Management Policies and Procedures"; and DoD Manual 5000.2- 
M, "Defense Acquisition Management Documents and Reports", all dated 23 February 
1991, replaced the 1987 versions of DoDD 5000.1 and DoDI 5000.2. Publication of 
these documents was the result of a two year project under three Under Secretaries of 
Defense for Acquisition. More than 60 previous directives, instructions, manuals and 
memoranda were canceled by these new documents. 

This new "5000 Series" provided a single uniform acquisition system for all of 
the Services’ acquisition programs: major defense acquisition programs, non-major 
defense acquisition programs, and highly sensitive classified programs. The DODI 
5000.2 states that the acquisition process shall be structured in discrete phases separated 
by major milestones. Under the old DoDI 5000.2, there were six major milestones (O 
through V), and five phases. The new DoDI 5000.2 provides for five major milestones 
with phases as shown in Figure l.[Ref. 11] 



Figure 1. Acquisition Milestones and Phases. 
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B. THE ACQUISITION CATEGORY SYSTEM 


Under the new DoD 5000 series, defense acquisition programs are divided into 
acquisition categories (ACATs) using the criteria shown in Figure 2. The AC AT 
designations indicate which programs come under the Defense Acquisition Board (DAB) 
or under a DoD Component Acquisition Executive for review and oversight. The DAB 
is the senior DoD acquisition review board and it is chaired by the Under Secretary of 
Defense for Acquisition. The Vice Chairman of the Joint Chiefs of Staff is the Vice- 
Chair. Other members of the Board are: the Deputy Under Secretary of Defense for 
Acquisition; Service Acquisition Executives of the Army, Navy, and Air Force; the 
Director of Defense Research and Engineering; the Assistant Secretary of Defense for 
Program Analysis and Evaluation; the Comptroller of the Department of Defense; the 
Director of Operational Test and Evaluation; the appropriate DAB Committee Chair; and 
the DAB Executive Secretary. The DoD Component Acquisition Executive is a single 
official within each DoD Component who is responsible for all acquisition functions 
within that Component. For the Army, this individual is the Assistant Secretary of the 
Army for Research, Development, and Acquisition. [Ref. 12] 

The programs with the lowest cost and risk, those that were classified as non¬ 
major under the old system, are classified as ACAT III and IV under the new series. A 
major distinction between ACAT III and IV programs and ACAT I and II programs is 
that the DoD Component Acquisition Executives may delegate decision authority for 
ACAT III and IV programs to the lowest level deemed appropriate within their respective 
organizations. [Ref. 13] For Army ACAT IV systems, this Milestone Decision Authority 
(MDA) designation is generally established at the Major Subordinate Command (MSC) 
level within the Army Material Command (AMC). See Figure 3 for a schematic of the 
applicable U.S. Army Command structure.[Ref. 14] 

Within U.S. Army practices, any program that does not meet the criteria for 
ACAT I or II, but has a Program or Product Manager assigned, receives an ACAT III 
designation. All other programs are designated as ACAT IV by default, and are managed 
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ACAT 


SELECTION CRITERIA 


DESIGNATION AUTHORITY 


MILESTONE DECISION 
AUTHORITY 




1 

• A program not classified as 
rtigniy sensitive by the Secretary 
oi Defense that has: 

** Seen designated by the 

Under Secretary of Defense 
(Acquisition) as an 
acquisition category ( 
program or is 

Estimated by the Under 
Secretary to require: 

- An eventual expenditure 
for research, development, 
test, and evaluation of more 
than $200 million in fiscal 
year i960 constant dollars 
(approximately S300 million 
in fiscal year 1990 constant 
dollars); or 

• An eventual expenditure 
for procurement of more 
than SI billion in fiscal year 
1980 constant dollars 
(approximately $1.8 billion 
in fiscal year 1990 constant 
dollars) 

• Under Secretary or Oefense 
(Acquisition) 

• Acquisition category 1 programs 
are further designated by the 
Under Secretary of Defense 
(Acquisition) as either requiring 
decision by the: 

•• Under Secretary - ACAT 1 D 

** Component Head - ACAT IC 

• aCATI 0 - Under Secretary of 
Qefeme (Acautsmon) 

• ACATIC - OoO Component Head 
or, if delegated, the OoO 
Component Acquisition 

Executive 

II 

• A program not meeting the 
criteria for category f that has: 

•• Seen designated by the OoO 
Component Head as an 
acquisition category II 
program oris 

Estimated by the OoO 
Component Head to require: 

- An eventual exoenditure 
for research, development, 
test, and evaluation of more 
than $75 million in fiscal 
year 1980 constant dollars 
(approximately SI IS million 
in fiscal year 1990 constant 
dollars); or 

• An eventual expenditure 
for procurement of more 
than $300 million in fiscal 
year 1980 constant dollars 
(approximately S540 million 
in fiscal year 1990 constant 
dollars) 

• OoO Component Head or. if 
delegated, the OoO Component 
Acquisition Executive 

• OoO Component Head or, i f 
delegated, the OoO Component 
Acquisition Executive 

111 

• Programs not meeting the 
cntena for category 1 and 11 that 
have been designated category 

III by the OoO Component 
Acquisition Executive 

• OoO Component Acquisition 
Executive 

• Lowest level deemed appropriate 
by the designation authority 

IV 

• All other acquisition programs 
for which the milestone decision 
authority should be delegated to 
a level below that required for 
category ill 

• OoO Component Acquisition 
Executive 

« Lowest level deemed aoproonate 
by the designation authonty 


Figure 2. The Acquisition Categories and Milestone Decision Authority, from Ref. 2 
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U.S. Army Organizational Structure 

ACAT IV Programs 
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Figure 3. Army ACAT IV Management Organizational Structure 















by individuals within the MSCs who do not receive a formal designation as Program or 
Product Manager. For programs in production, these ACAT IV managers are typically 
civilian Weapons System Managers at the MSC headquarters. For those programs still 
in development, the manager is typically a civilian technical expert with an informal title 
of Project Officer. He is usually located at a Research, Development, and Engineering 
Center (RDEC) that is subordinate to a related MSC.[Ref. 15] 

A typical example of an ACAT IV program would be the Army’s next generation 
of general purpose tentage, the Modular General Purpose Tent System. This effort seeks 
to apply modem materials and structures to replace the 1950s vintage tentage currently 
in use by all of the Services. The technological risk is very low, total development costs 
will be less than $2 million, and the requirement is not threat or vulnerability driven. 
This logistics/quality-of-life oriented effort certainly does not warrant the oversight 
required for a major ACAT I or II weapon system. The management structure supporting 
this effort includes the GS-12 civilian Project Officer and his management chain-of- 
command. The MDA is the MSC, Aviation and Troop Command (ATCOM), Executive 
Director for Integrated Materiel Management. [Ref. 16] 

C. STREAMLINING THE PROCESS 

One aspect of the new DoD 5000 series that did not exist in the old regulations 
is that the Services are now prohibited from regulatory supplementation. DoD 5000.2 
states: 

Unless prescribed by statute or specifically authorized herein, the 
policies and procedures set out in this Instruction shall not be 
supplemented without the prior approval of the Under Secretary of 
Defense for Acquisition.... Implementation directives, instructions, 
regulations, and related issuances shall be kept to the essential minimum 
as deemed appropriate by the DoD Component Acquisition Executive. 

Copies of all such issuances shall be provided to the Director of 
Acquisition Policy and Program Integration, Office of the Under Secretary 
of Defense for Acquisition within 10 days of publication. [Ref. 17] 
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This effectively ensures that the Services do not circumvent the goal of the new 
regulation by instituting Service-specific implementation policy. Unfortunately, it also 
creates an inability for a Service to implement beneficial across-the-board policies 
without OSD approval. Streamlining the procedures and processes of DoD 5000 for all 
ACATIV programs, for instance. DoD 5000 does, however, encourage streamlining or 
tailoring of the process on a project-by-project basis. DoD 5000.2 states: 

Tailoring of Acquisition Procedures and Documentation. The 
policies and procedures described in this Instruction shall apply directly 
to Acquisition Category I programs and will be tailored as defined in 
subsection B.5., above, for acquisition category II, III, and IV programs 
subject to the approval of the milestone decision authority.[Ref. 18] 

Furthermore, DoD 5000.1 requires that acquisition strategies and program plans 
be streamlined to minimize acquisition time and life-cycle costs, consistent with the 
urgency of the need and degree of technical risk involved. Streamlining becomes 
particularly important for the smaller, ACAT IV programs since the procedures outlined 
in DoD 5000 are written for the management oversight needs of the largest ACAT I and 
II weapon systems. [Ref. 19] 

As noted in the above excerpt from DoD 5000.2, DoD 5000 gives broad power 
to the MDA to decide, among other things, whether a program streamlining plan is 
acceptable. This power was most recently emphasized by the Under Secretary of Defense 
(Acquisition and Technology) in a 23 August 1994 memorandum to the Service 
Secretaries titled: "Tailoring Acquisition Procedures and Documentation for Acquisition 
Category II, III and IV Programs" which should allow for consideration of more 
innovative streamlining methods than could be agreed to by all members of the 
acquisition team in the past. The memorandum states: 

The specific form or number of program documents should be determined 
by the MDA. As long as tailoring is consistent with any applicable 
statutory requirements, the MDA has full authority to reduce or eliminate 
any procedures or documents that he or she deems unnecessary.[Ref. 20] 
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As defined in the Defense Systems Management College’s Risk Management 
Concepts and Guidance Manual, streamlining is a part of the risk management decision 
making process, a necessary consideration if the manager is to optimize his project’s 
cost, schedule and performance. While developing a program plan or acquisition strategy, 
as well as during the life of a program, the manager must consider risk-related trade-offs, 
some of which involve streamlining considerations. Risks to the manager are all rooted 
in the determination to deliver a specified product or level of performance at a specified 
time for a specified cost. The manager may fail in any of three ways, or combination 
thereof. The product may not be up to the performance level specified, the actual costs 
may be too high, or delivery may be too late. According to the Risk Management 
Manual, risk can be categorized into five facets to segment and manage the cost, 
schedule, and performance issues of a project. These risk facets are technical 
performance, supportability, programmatic (loss of funds, availability of test assets, etc.), 
cost, and schedule. If streamlining isn’t considered, projects run the risk of being too 
slow to fielding, or overly costly to develop. Streamlining will increase risk, but the goal 
of any effective manager is to balance the project’s risk with the cost, schedule, and 
performance attributes of the effort. Thus, streamlining is a large part of the manager’s 
overall risk management strategy. [Ref. 21] 

Illustrative examples of this risk management process as it relates to streamlining 
and the schedule, performance, and cost attributes of a program are: (1) Schedule: An 
urgent mission need for a capability, either because of an increase to combat 
effectiveness or due to system obsolescence, can lead to a need for streamlining of the 
process in order to shorten the time to fielding for the new capability. The manager 
evaluates this requirement, the schedule reduction options available, and the risk 
associated with each; (2) Performance: The decision-maker determines that a program’s 
relatively low degree of technical risk, based on the maturity of the technology being 
employed and the complexity of the system, will allow for skipping of milestone events 
and streamlining of the program; or (3) Cost: The decision maker considers that 
streamlining the development program will reduce costs in both development and 


16 



operations and support (where the capability being replaced may have a higher 
maintenance cost due to it being beyond its useful life). He must evaluate cost concerns 
with an acceptable degree of risk. 

All of these streamlining considerations are based on trade-offs of costs (e.g., lack 
of availability of development funds, potential O&S cost savings) and schedule (e.g., 
urgent need) with the risk of technical performance, supportability, or programmatic 
failure with each of these decisions. To properly assess these considerations, the decision 
maker requires a sound risk assessment methodology utilizing data concerning the 
impacts of each decision option and the key factors relating to each. Key factors (utilized 
for trade-offs among schedule, cost, and risk) would be attributes such as technological 
maturity, system complexity, urgency of need, availability of funding (both for 
development and production), and availability of personnel and facilities (for oversight, 
testing, etc.). This study seeks to establish this assessment data and key factors for use 
by AC AT IV managers in conducting risk assessments on their programs. [Ref. 22] 

The researcher’s experience and information from the MSC POCs identified the 
streamlining techniques employed for ACAT IV projects that were investigated in this 
research effort. Those techniques are summarized as follows: 

1. Combine Milestones 0 and I (Skip Concept Exploration (CE) Phase) 

2. Combine Milestones I and II (Skip Demonstration/Validation (Dem/Val) 

Phase) 

3. Combine Milestones I and III (Skip Dem/Val and Engineering & 

Manufacturing Development (EMD) Phases) 

4. Combine Dem/Val Development Test (DT) and Operational Test (OT) 

5. Conduct concurrent (overlapping) Dem/Val DT and OT 

6. Combine EMD DT and Initial Operational Test & Evaluation (IOT&E) 

7. Conduct concurrent (overlapping) EMD DT and IOT&E 

8. Conduct Customer Tests in lieu of a formal Development Test in Dem/Val 

9. Conduct Customer Tests in lieu of a formal Development Test in EMD 

10. Conduct user evaluation in lieu of formal IOT&E in EMD Phase 

11. Request Milestone Decision Authority delegate Milestone Decision Review 

Chairmanship to RDEC Management 

12. Request DA-Directed Procurement prior to Type Classification-Standard 
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13. Staff Milestone Decision Package prior to receipt of Independent 
Evaluation/Assessment Reports (IER/IAR), with reports expected prior to the 
decision meeting 

14. Use Interim Assessment or Evaluation Reports to support a Milestone 
Decision, with decision contingent upon favorable Final Reports 

15. Use Independent Evaluator Briefing in lieu of written reports to support a 
Milestone Decision, with written report(s) to follow 

16. Waive requirement for a Cost and Operational Effectiveness Analysis 
(COEA) 

17. Waive requirement for a separate System Threat Assessment Report (STAR) 

18. Waive requirement for a separate Logistics Demonstration 

19. Utilize Test Integration Working Group (TIWG) as Electromagnetic 
Environmental Effects (E3) Requirements Board 

20. Conduct abbreviated Logistics Support Analysis and Reporting (LSA/LSAR) 

21. Don’t prepare separate Configuration Management Plan 

22. Don’t generate Independent Life Cycle Cost Estimate 

23. Obtain Milestone Decision Authority approval to solicit a contract prior to 
Milestone Decision Review 

24. Utilize a single contract for Dem/Val and EMD 

25. Utilize a single contract for EMD and First Production 

26. Conduct production First Article Tests to validate development deficiency 
corrections 

27. Validate technical manuals/support system during first production instead of 
during EMD 

Most of these 27 techniques are self-explanatory. The few that require further 
explanation are described as follows: 

Techniques 8 and 9 - Conduct Customer Tests in lieu of a formal DT in 
Dem/Val and EMD. Here, Customer Tests are defined as tests conducted by the 
development tester. Test and Evaluation Command (TECOM), that are based solely on 
the direction provided by the materiel developer. For these techniques, Customer Tests 
do not refer to those conducted with operational testing issues. The materiel developer 
is the customer of TECOM in this case. This differs from formal DT where the basis for 
the test is a requirements document. Customer tests are typically used in early 
development to evaluate concepts and they are usually less expensive than comparable 
DT because of lower instrumentation and oversight requirements. However, the results 
may not meet the needs of the user. 
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Technique 10 - Conduct user evaluation in lieu of formal IOT&E in EMD Phase. 
A user evaluation is defined as use of a developmental system by real field units in an 
actual field exercise, with non-intrusive methods of data collection such as interviews 
following the exercise. Formal IOT&E, on the other hand, is defined as use of a 
developmental system by representative soldiers in a field exercise created specifically 
to evaluate the performance of the new system. IOT&E is substantially more costly than 
a user evaluation, but it can also result in substantially more useful data on the 
performance of a developmental system. 

Technique 12 - Request DA-Directed Procurement prior to Type Classification- 
Standard. In cases where fielding of a developmental system is urgently required and the 
development process has not been completed, DA has the option to direct the immediate 
procurement of the item. This would then be followed by completion of formal 
development and the retrofitting of any required changes into the previously fielded 
systems. 

Technique 19 - Utilize Test Integration Working Group (TIWG) as 
Electromagnetic Environmental Effects (E3) Requirements Board. The requirement for 
an E3 Requirements Board for all Army development programs was recently directed by 
AMC. Since the agencies and individuals necessary for both the E3 Board and the 
already existing TIWG are almost identical, combining of the two would save on the 
number of separate committees involved in oversight of a program. 

Technique 26 - Conduct production First Article Tests to validate development 
deficiency corrections. For most development programs, any serious system deficiencies 
uncovered during EMD must be corrected, and the corrections validated before the Army 
is willing to enter production. The validation could consist of an extra evaluation/test 
prior to conducting the Milestone III Decision Review. First Article Testing validates the 
adequacy of production units before allowing full rate production to begin, using the first 
units produced by the production contractor. It usually retests many of the same functions 
as would be required to validate development deficiency corrections. Therefore, if the 
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MDA is willing to accept the Milestone III Decision prior to the validation of the 
deficiency corrections, they could be validated during FAT. 

A hypothetical example of streamlining of an ACAT IV program would be as 
follows: The manager may consider the technical risk of the system that is to be 
developed to be low enough to allow for a combined Milestone I and II, moving the 
project directly from Concept Exploration (CE) into the Engineering and Manufacturing 
Development (EMD) phase. The risk of this approach is that the system will not be 
sufficiently mature for EMD, resulting in rework and delays and/or less than desired 
performance. But the manager must balance this risk with the need to field the new 
system by a prescribed date (for instance, driven by the end of the useful life of the 
existing system). Streamlining provides the best management plan given the urgency of 
the schedule and the relatively low risk. 

D. CHAPTER SUMMARY 

The new DoD 5000 series of acquisition management regulations concentrates on 
the management oversight required for large ACAT I programs. The regulations outline 
the processes required of ACAT I managers, and encourages tailoring, or streamlining 
of the process by ACAT II, III, and IV managers, while not allowing for across-the- 
board Service-level policies. Individual managers need to streamline to effectively 
manage their program’s risk, but they lack key data necessary for conducting proper risk 
assessments. This study seeks to provide this missing risk assessment data. 
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ffl. METHODOLOGY 


The methodology utilized for this effort was a management problem analysis using 
both quantitative and qualitative techniques. The qualitative techniques involved the use 
of personal and telephonic interviews to determine the opinions of the acquisition policy 
Subject Matter Experts from each of the AMC Commands concerning the validity and 
breadth of the techniques to be investigated. The information from this process was 
assimilated into a broad-based questionnaire that was distributed Army-wide to all AC AT 
IV development managers, with questions that required numeric and essay responses. 
The numeric responses were quantitatively analyzed, and the essay responses were used 
to find elaboration/causal relationships for specific responses and/or trends observed in 
the data. 

A. RESEARCH PROCESS STEPS 

1. Telephonic interviews were conducted with individuals within AMC, the 
Department of Army Headquarters, and the AMC MSCs to develop a list of 
MSC/RDEC acquisition policy points of contact (POCs) for ACATIV programs. 
The individuals identified were either the Command’s overall acquisition policy 
POC, or a senior manager charged with management of all ACAT IV programs 
for that Command. The individuals are listed in Appendix A. 

2. Telephonic interviews were conducted to: 

a. Develop a valid list of streamlining techniques used by ACAT IV managers, 
for use as a basis for questionnaires to ACAT IV managers. 

b. Identify ACAT IV project managers’ names and addresses to survey. 
Development of this list was necessary because ACAT IV programs are not all 
tracked by AMC. Therefore, the only way to obtain a comprehensive list was to 
call into each Command and search out an individual or individuals 
knowledgeable on the subject who had access to the names and addresses of the 
ACAT IV managers. A total of seventy-seven individuals were identified as 
ACAT IV managers. For all Commands except the Missile Command (MICOM) 
and the Aviation Applied Technology Directorate (AATD) of the Aviation and 
Troop Command, specific names and addresses were identified. For MICOM and 
AATD, since only one senior manager within each Command was in charge of 
all ACAT IV programs, the POCs noted above agreed to distribute the 
questionnaires. 

3. Using the streamlining techniques previously listed, and the U.S. Army 
Research Institute for the Behavioral and Social Sciences Questionnaire 
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Construction Manual, an appropriate survey questionnaire for the study was 
constructed. See Section B for the specifics of the questionnaire development 
process and Appendix B for a copy of the questionnaire. [Ref. 23] 

4. Draft questionnaires were provided to the MSC/RDEC acquisition policy POCs 
for their review and comment. Since these individuals are their Command’s 
acquisition policy "experts", their responses helped refine the document and 
served as a check that the questionnaire was complete and understandable before 
surveying their AC AT IV managers. The POCs were not surveyed further. 

5. The questionnaires were sequentially numbered for control purposes and 
mailed to each of the AC AT IV managers that were previously identified. The 
mailing included a personally addressed and individually signed cover letter 
requesting participation, stating the purpose of the study, and articulating the 
voluntary, confidential and not-for-attribution nature of the study. Additionally, 
each package contained a return addressed, postage paid, Government reply 
envelope. Four weeks were allowed for responses. 

6. Finally, the data was compiled and analyzed as discussed in Chapter V. 


B. QUESTIONNAIRE DEVELOPMENT 

A quantitative questionnaire with provisions for subjective comments, reproduced 
in Appendix B, was prepared as the survey instrument. It is structured in two parts. The 
first part focuses on the size of the projects managed by the respondent, to allow for 
categorization of responses by size. The second part addresses the respondents opinions 
regarding DoD 5000 streamlining techniques. The questionnaire structure chosen was an 
ordinal, Likert scale, with compound questions: a binary set and two five point adjectival 
questions. The following scale was employed for the adjectival questions regarding cost 
savings and schedule reduction potential. This non-linear scale was chosen to be able to 
differentiate among the levels of savings for the lowest, as well as the highest, saving 
techniques: 


Rating 1 

Cost Savings $0-$100 

Schedule Reduction 0-7 days 


2 3 

$100-$1000 $1K-$10K 

1-4 wks 1-6 mos 


4 5 

$10K-$100K >$1(XK 
6-18 mos >18 ms 
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From the referenced questionnaire manual, these selections best fit the type of 
questions and data being addressed in this research effort. For example, the reference 
states that ordinal Likert scales are the preferred and most widely used type when 
multiple choice responses are required and the distance between each scale point isn’t 
assumed to be equal. In addition, it is the preferred scale when the categories are 
mutually exclusive and exhaustive, as the list of streamlining initiatives should be. Five 
point scales are the most common and within the range of breakouts that yields roughly 
equal responses. The scale will be fully labeled, as this is noted in the reference to 
provide less skewed results. This should provide an understandable and statistically useful 
range of options. [Ref. 24] 

The questionnaire addressed the research questions as outlined in the following 
dendritic process: 

Subsidiary Research Question 1: How do Army Commands streamline ACAT IV 
programs differently? 

Criteria: For each Command, what are the most frequently used streamlining 
techniques? 

Measure of Effectiveness: For each Command, what techniques are used? 

Data Requirement: Techniques used by each Command. 

Data Source: Survey Question a. 

Measure of Effectiveness: For each Command, what is the frequency of use of 

each technique? 

Data Requirement: Number of times each technique is used by each 

Command. 

Data Source: Survey Question a. 
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Criteria: What factors cause each Command to use certain techniques (as compared to 
others)? 

Measure of Effectiveness: For the AC AT IV managers in each Command, why 
was a technique chosen? 

Data Requirement: Manager comments on factors leading to a decision. 
Data Source: Survey Question a. comments. 

Subsidiary Research Question 2: How do the various streamlining techniques compare 
with respect to program cost savings? 

Criteria: For each technique, what is the program cost savings? 

Measure of Effectiveness: For each size program what is the distribution of cost 
savings ratings among the streamlining techniques? 

Data Requirement: Ratings of cost savings associated with each 
technique. 

Data Source: Survey Question b. 

Criteria: What factors cause each manager to utilize certain techniques (as compared to 
others)? 

Measure of Effectiveness: For each manager, why was a rating chosen? 

Data Requirement: Manager comments on factors leading to a decision. 
Data Source: Survey Question b. comments. 

Subsidiary Research Question 3: How do the various streamlining techniques compare 
with respect to program schedule reduction? 

Criteria: For each technique, what is the program schedule reduction? 

Measure of Effectiveness: For each size program what is the distribution of 
schedule reductions ratings among the streamlining techniques? 

Data Requirement: Ratings of schedule reductions associated with each 
technique. 

Data Source: Survey Question c. 
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Criteria: What factors cause each manager to utilize certain techniques (as compared to 
others)? 

Measure of Effectiveness: For each manager, why was a rating chosen? 

Data Requirement: Manager comments on factors leading to a decision. 

Data Source: Survey Question c. comments. 

The survey conducted utilizing the questionnaire, as part of this study’s 
methodology, resulted in quantitative and narrative data concerning the streamlining 
techniques being investigated. In the next chapter the data will be reviewed, tabulated and 
presented for analysis. 
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IV. DATA PRESENTATION 


In support of this thesis, 74 questionnaires were distributed to the ACAT IV 
development managers at various U.S. Army Commands to inquire about their use of 
program streamlining techniques for ACAT IV development programs. Initially there 
were 16 responses. After follow-up inquiries, it was discovered that 35 individuals were 
inappropriate survey subjects, so the number of potential respondents was reduced to 39 
and a final total of 19 responses were obtained. Some Commands provided limited or no 
responses. 

The two Commands that did not respond were CECOM and MICOM. Since these 
two cases of nonresponsiveness appeared to represent systemic problems, the Command 
ACAT IV acquisition policy POCs (Appendix A) were queried about the lack of 
responses, instead of directly contacting individual ACAT IV managers. The discussions 
with these POCs revealed a difference with the way smaller-sized development programs 
are managed for the commodities associated with CECOM and MICOM. This difference 
was not readily apparent during earlier discussions. Specifically, both Commands use 
managers who are not certified Program Managers to conduct the early development 
phase of programs, which, as pointed out earlier, automatically categorizes them as 
ACAT IV. What is unique to these two Commands is that all of their programs transition 
to ACAT III, or higher status early in development by having management responsibility 
transferred to a certified Program Manager (PM) within the Program Executive Officer 
(PEO) chain-of-command. This also happens for some but not all programs at the other 
AMC Commands. Therefore, the ACAT IV managers within CECOM and MICOM do 
not get very involved in the DoD 5000 policies, most of which are not applicable early 
in the development process, and the managers had no experience base with which to 
respond to the survey. The CECOM POC stated: 

The survey you supplied seems to be directed more to the type of 
acquisition done at Natick. Here at CECOM, a program goes through the 
early stages of development, funded by 6.2, 6.3 R&D funds. When a 
program is ready for 6.4 funding (the EMD phase), it transitions to a PEO 
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portfolio.... ACAT IV, by definition, means the system is managed by 
other than a board selected Program Manager. Therefore, a system that 
starts as an ACAT IV can then become an ACAT III when it transfers to 
a PM under a PEO. 

Given this difference, these two Commands fall outside of the scope of the survey 
since it keyed on ACAT IV development managers and the policies of DoD 5000. 
Thereby, the number of potential survey respondents is appropriately reduced to 39. 
However, the difference in management philosophies between these two Commands and 
the other four is of interest, and the Analysis Chapter includes a comparison based on 
the information revealed in the study and the researcher’s experience. 

Follow-up discussions with non-responsive managers from the other Commands 
revealed a number of points regarding the questionnaire and the researcher’s 
assumptions. Assumptions concerning the managers’ willingness to participate and their 
knowledge of the DoD 5000 development process were not valid for all ACAT IV 
managers. Also, a number of managers did not consider themselves qualified to respond 
due to their lack of experience with the entire process (which can take long enough that 
many managers have not experienced all or much of it in their careers), and/or lack of 
knowledge of the processes being proposed for streamlining. Some of the individuals 
contacted were not familiar with the DoD 5000 terminology, and this created a barrier 
to their understanding of the proposed streamlining techniques. 

Given the above, the data that was received is deemed to be valid. The 
respondents are still assumed to be knowledgeable and any conclusions drawn valid. A 
limitation is created as it will be difficult to establish statistical significance for the 
results, since the quantity of observations is lower than expected. Therefore, the analysis 
will be conducted on a qualitative, rather than predominately quantitative, basis. 

A. DATA TABULATION 

As the questionnaire were received from the respondents, the data was loaded in 
two databases. The quantitative responses to the survey were tabulated in a Minitab 
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database and the narrative responses were tabulated by question number in a WordPerfect 
text database. The raw data is available in Appendices C and D for the quantitative and 
narrative data respectively. 

B. DATA PRESENTATION 

For the first research sub-question (How do Army Commands streamline ACAT 
IV programs differently?), the Command-by-Command comparison, the quantitative 
responses are summarized in Table 1. For each Command that was queried in the survey, 
the table depicts the total number of respondents, as well as the number and proportion 
of the respondents who responded positively regarding their awareness of each of the 
streamlining techniques. Note the low number (two) of respondents from both AATD and 
ERDEC, as well as the lack of responses from AATD, ERDEC, and TARDEC (zeros 
in the ”# Yes" and " % Yes" columns) for a number of techniques. One aspect of the 
Analysis Chapter will be investigation of each of these non-response areas, to ascertain 
whether the Commands were aware of the technique or not. 

For the second and third research sub-questions (How do the various streamlining 
techniques compare with respect to program cost savings?) and (How do the various 
streamlining techniques compare with respect to program schedule reduction?), the 
Technique-by-Technique comparisons, the quantitative responses are summarized in 
Table 2. The five categories for total program RDTE cost, and the seven categories for 
development durations, established in the survey were consolidated in the table into two 
categories (small and large programs), to provide clarity for evaluation. This was 
necessary because the number of categories established during questionnaire development 
became inappropriate, due to the small number of survey responses in each category. 
Large programs were defined for purposes of the analysis as those requiring $7 million 
or more of RDTE funding (for the cost comparisons) and 3 years or more to complete 
development (for the schedule comparisons). For each size program, the table depicts the 
total number of respondents for both cost savings and schedule reduction, as well as the 
number of respondents who replied regarding each technique and an average of the 
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ACAT IV Streamlining Techniques 
Frequency of Use by Command 

Command 

Natick 

AATD 

ERDEC 

RSI 

No. of 









Respondents 

9 


2 


2 


6 


Streamlining 

Technique 

# Yes 

% Yes 

# Yes 



% Yes 

# Yes 

% Yes 

1 

7 

0.8 

2 

1.0 

1 

0.5 

4 

0.7 

2 

9 

1.0 

2 

1.0 

2 

1.0 

3 

0.5 

3 

6 

0.7 

0 

0.0 

2 

1.0 

2 

0.3 

4 

9 

1.0 

0 

0.0 

2 

1.0 

4 

0.7 

5 

5 

0.6 

2 

1.0 

2 

1.0 

3 

0.5 

6 

5 

0.6 

1 

0.5 

2 

1.0 

1 

0.2 

7 

3 

0.3 

2 

1.0 

2 

1.0 

2 

0.3 

8 

5 

0.6 

1 

0.5 

2 

1.0 

0 

0.0 

9 

3 

0.3 

0 

0.0 

2 

1.0 

0 

0.0 

10 

6 

0.7 

2 

1.0 

2 

1.0 


0.2 

11 

7 

0.8 

1 

0.5 

0 

0.0 

3 

0.5 

12 

4 

0.4 

1 

0.5 

0 

0.0 

0 

0.0 

13 

8 

0.9 

2 

1.0 

2 

1.0 

4 

0.7 

14 

7 

0.8 

2 

1.0 

2 

1.0 

4 

0.7 

15 

5 

0.6 

1 

0.5 

1 

0.5 

3 

0.5 

16 

7 

0.8 

2 

1.0 

1 

0.5 

2 

0.3 

17 

6 

0.7 

1 

0.5 

2 

1.0 

3 

0.5 

18 

8 

0.9 

0 

0.0 

2 

1.0 

2 

0.3 

19 

4 

0.4 

2 

1.0 

1 

0.5 

2 

0.3 

20 

3 

0.3 

1 

0.5 

2 

1.0 

2 

0.3 

21 

3 

0.3 

1 

0.5 

2 

1.0 

0 

0.0 

22 

2 

0.2 

0 

0.0 

1 

0.5 

1 

0.2 

23 

6 

0.7 

1 

0.5 

1 

0.5 

1 

0.2 

24 

8 

0.9 

1 

0.5 

2 

1.0 

3 

0.5 

25 

7 

0.8 

1 

0.5 

2 

1.0 

1 

0.2 

26 

6 

0.7 

2 

1.0 

2 

1.0 

2 

0.3 

27 

4 

0.4 

2 

1.0 

2 

1.0 

3 

0.5 


Table 1. Frequency of Use by Command 
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Projected Savings by Program Size 
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Table 2. Projected Savings by Program Size 









savings ratings provided by these respondents. In order to qualitatively evaluate the 
results of the survey, the ratings of the individual respondents were averaged to: (1) 
obtain small to large program trends in the savings associated with the various 
techniques, and (2) establish which techniques have the very highest and lowest savings 
(in terms of cost and schedule) for small and large programs. The average ratings do not 
represent expected savings. The scale is not linear and the response rates are too low to 
provide confidence in the point estimates. The rating scale used in the survey is provided 
at the bottom of the table for reference. 

The following Analysis Chapter utilizes the data in Tables 1 and 2 to investigate 
differences between the usage levels of Natick and a combination of the other 
Commands. In addition, differences between the cost savings and schedule reduction 
potential of small and large programs are analyzed. 
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V. ANALYSIS 


This analysis is based on comparisons of the most prevalent streamlining 
techniques and those that had the highest and lowest savings, with respect to cost and 
schedule. The lowest saving techniques are being evaluated because a number of them 
were both low in savings and high in use. This indicates that they may be valuable for 
other reasons, or their implementation is so easy as to be routine. The narrative 
responses are used to provide "cause-and-effect" reasons for similarities and differences 
among Commands and among techniques (in terms of their frequency of use and the 
expected level of cost savings and schedule reduction). 

A review of the narrative comments regarding three of the techniques led to their 
subsequent elimination for further analysis. The three techniques are 

Technique 8. Conduct Customer Tests in lieu of a formal Development 

Test in Dem/Val 

Technique 9. Conduct Customer Tests in lieu of a formal Development 

Test in EMD 

Technique 23. Obtain Milestone Decision Authority approval to solicit a 

contract prior to Milestone Decision Review 

The first problem, regarding Techniques 8 and 9 (Conducting Customer Tests in 
lieu of formal DT) concerned varying definitions for the term "customer test". The 
intended definition was a TECOM test conducted without formal documentation and 
planning, where the materiel developer is the customer. Many respondents from 
Commands other than Natick responded as if "customer testing" meant "user testing", 
which was addressed in Technique 10 (Conduct user evaluation in lieu of formal IOT&E 
in EMD phase). The varying interpretations of the definition of customer testing makes 
any comparisons based on these techniques invalid. For Technique 23 (Obtain MDA 
approval to solicit a contract prior to Milestone Decision Review) the narrative comments 
from the survey participants indicated that some respondents misread the technique and 
they addressed it as if it proposed awarding the contract, vice soliciting it. The question 
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did not intend to address the award of a contract prior to the Review. Given this, the 
responses are not clearly differentiated and the information is not clear. Therefore, the 
analysis ignores Techniques 8, 9, and 23. 

A. COMMAND-BY-COMMAND COMPARISON 

1. Command Awareness of Techniques 

The first Measure of Effectiveness addressed the question of whether all the 
Commands were aware of all of the techniques. From the data in Table 1, there are 
seven techniques that were not considered by any of the respondents for at least one 
Command. 


Technique 3. 

Technique 4. 

Technique 11. 

Technique 12. 

Technique 18. 
Technique 21. 
Technique 22. 


Combine Milestones I and III (Skip Dem/Val and 
Engineering & Manufacturing Development (EMD) Phases) 
Combine Dem/Val Development Test (DT) and Operational 
Test (OT) 

Request Milestone Decision Authority delegate Milestone 
Decision Review Chairmanship to RDEC Management 
Request DA-Directed Procurement prior to Type 
Classification-Standard 

Waive requirement for a separate Logistics Demonstration 
Don’t prepare separate Configuration Management Plan 
Don’t generate Independent Life Cycle Cost Estimate 


Because the questionnaires were targeted to specific individuals, the non-responses 
concerning these techniques may not indicate that the Command was not aware of the 
technique, but only that the individuals responding were not. In addition, some 
individuals who did not check the survey block to indicate that they were aware of a 
technique did, however, provide narrative comments regarding the same technique. Given 
this, a review of the survey narrative comments for these seven techniques revealed the 
following specific information: 

For Techniques 3 (Combine Milestones I and III) and 4 (Combine Dem/Val DT 
and OT), AATD did not have any positive responses. However, evaluation of their 
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narrative responses revealed that they were aware of the techniques and that they have 
sometimes used them. Specific comments were: 

Technique 3 - Off-the-shelf or low risk technology are ideally suited for 
combining I and III for an AC AT IV program. 

Technique 4 - Off-the-shelf items should not need very much DT/OT 
testing because of the low technology risks. 

For Techniques 11 (Request MDA delegate Milestone Decision Review 
Chairmanship to RDEC management), 12 (Request DA-Directed Procurement prior to 
Type Classification-Standard), 21 (Don’t prepare separate Configuration Management 
Plan), and 22 (Don’t generate Independent Life Cycle Cost Estimate) various Commands 
did not have any positive responses, but their narrative comments revealed an awareness 
of the technique. In addition, their comments expressed disagreement with use of the 
technique as the reason they did not respond positively to the survey question. Comments 
included: 

Technique 11.- Like having the General Officer sign it. 

Technique 21. - Configuration Management Plan must be specific to 
parties involved. 

For Technique 18 (Waive requirement for a separate Logistics Demonstration) 
AATD did not have any positive responses. A review of their narrative responses 
indicated agreement with the intent of the technique, but there was no indication of 
whether or not they were aware of the idea prior to the survey. 

In general, for the seven techniques that were not considered by any of the 
respondents from a particular Command, all of the individuals who did not respond 
provided narrative comments and in all but one case, the Command was aware of the 
technique. Negative responses were due to disagreement with the use the technique, not 
lack of awareness of the technique. The one technique where the narrative responses left 
it unclear as to whether AATD was aware of the technique (Waiving the Logistics 
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Demonstration), remains unknown. However, out of five Commands and 25 techniques 
(125 categories) only one category remained unresolved as to the Command’s awareness 
of a technique. 

All of the Commands are aware of all of the identified, available techniques. Any 
methods of cross-fertilization transmission are unknown. They may be independently 
concluding the same things, or they could be receiving suggestions from acquisition team 
members who are members of teams from different Commands (such as TRADOC 
schools, TECOM, AMC HQ, etc.). Regardless of the method being employed, the 
researcher’s concern that cross-fertilization was not taking place and certain Commands 
were unaware of specific streamlining techniques has proven to be invalid. 

2. Highest Usage Techniques 

To evaluate the second Measure of Effectiveness, comparing usage levels across 
techniques and Commands, the Natick responses were compared to those of all other 
Commands combined (labeled "non-Natick” throughout this document), see Table 3. As 
a result of combining non-Natick responses, the analysis of Natick operations is more 
thorough than any other Command. This was felt to be the most useful approach to data 
analysis, due to the limited responses by the non-Natick Commands. First, an overall 
comparison of Natick to non-Natick responses will be conducted, followed by a more in- 
depth evaluation of the techniques with the highest response frequencies for Natick and 
non-Natick. As a result, a number of interesting correlations are revealed. 

Overall Comparison - Figure 4 compares the frequency of use established for 
each technique for Natick and non-Natick respondents. The chart reveals general 
agreement between the two groups, with a consistently slightly higher usage for the 
Natick group. One possible reason for the consistent difference may be a bias created 
during the establishment of the streamlining techniques for the study. The researcher is 
from Natick and a number of the techniques listed in the survey were posed as a direct 
result of acquisition policy issues recently discussed at Natick, and, therefore, the Natick 
respondents may have been more familiar with the wording in the survey. 


36 



ACATIV Streamlining Techniques 
Frequency of Use 



Natick 

Other Commands' 
Total 

No. Of 





Respondents 

9 


10 


Streamlining 

Technique 

# Yes 

% Yes 

# Yes 

% Yes 

1 

7 

0.8 

7 

0.7 

2 

9 

1.0 

7 

0.7 

3 

6 

0.7 

4 

0.4 

4 

9 

1.0 

6 

0.6 

5 

5 

0.6 

7 

0.7 

6 

5 

0.6 

4 

0.4 

7 

3 

0.3 

6 

0.6 

8 

5 

0.6 

3 

0.3 

9 

3 

0.3 

2 

0.2 

10 

6 

0.7 

5 

0.5 

11 

7 

0.8 

4 

0.4 

12 

4 

0.4 

1 

0.1 

13 

8 

0.9 

8 

0.8 

14 

7 

0.8 

8 

0.8 

15 

5 

0.6 

5 

0.5 

16 

7 

0.8 

5 

0.5 

17 

6 

0.7 

6 

0.6 

18 

8 

0.9 

4 

0.4 

19 

4 

0.4 

5 

0.5 

20 

3 

0.3 

5 

0.5 

21 

3 

0.3 

3 

0.3 

22 

2 

0.2 

2 

0.2 

23 

6 

0.7 

3 

0.3 

24 

8 

0.9 

6 

0.6 

25 

7 

0.8 

4 

0.4 

26 

6 

0.7 

6 

0.6 

27 

4 

0.4 

7 

0.7 


Table 3. Frequency of Use, Natick vs. Other Commands Total 

















ACAT IV Streamlining Techniques 

Frequency of Use by Command 



Figure 4. Frequency of Use by Command 



The following is a review of each of the techniques with the highest differences, 
analyzing the differences between the two groups: 

For Technique 4 (Combine Dem/Val DT and OT) the non-Natick respondents 
expressed lower usage for three reasons: (1) they disagreed with the use of the technique 
because of its high risk and they felt that independent operational tests were necessary 
to remove developer biases, (2) they considered it too difficult to accomplish, citing 
problems with obtaining agreement from testers and evaluators, and (3) commodity 
specific concerns (such as the artillery ammunition not typically requiring OT and the 
man-firing of munitions prior to accomplishing significant DT). Note that non-Natick 
usage was still fairly high (it was a high usage technique for both groups). 

Technique 11 (Request MDA delegate Milestone Decision Review Chairmanship 
to RDEC management) is broadly applied at Natick. The non-Natick respondents 
expressed lower usage for two reasons: (1) they disagreed with the use of the technique 
because they preferred that a General Officer sign the minutes/decision, or because they 
did not consider the developer unbiased enough to chair the Review, and (2) the recent 
establishment of the new Command structure within AMC (including the consolidation 
of ARDEC and portions of Belvoir RDEC as parts of TARDEC) left it as an unresolved 
issue for some respondents at this point. One possible reason for its high usage at Natick 
is that the MDA is remotely located at ATCOM in St. Louis. However, Natick is not 
unique in being remotely located from the MDA. Both AATD and the parts of TARDEC 
that were surveyed are also located away from their MDAs. One unique organizational 
difference is that the Natick MDA at ATCOM is predominately concerned with aviation 
items. The lack of direct involvement in the items under development at Natick could 
provide an explanation for the high interest in having the Review Chairmanship at 
Natick. Interestingly, the recent change that places Natick under a new MSC, Soldier 
Support Command (SSCOM), which is co-located at Natick, may change this opinion to 
one expressed by a respondent from ERDEC: "Like having General Officer sign it". 

Technique 18 (Waive requirement for a separate Logistics Demonstration) is a 
program requirement monitored and enforced by the Army Materiel Systems Analysis 
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Activity (AMSAA). According to the narrative comments, for non-Natick commodities, 
the demonstration is not required in many cases (for ammunition for instance). 

For Technique 25 (Utilize a single contract for EMD and First Production) the 
non-Natick respondents disagreed with the use of the technique. Some of the comments 
were: 


Cost factors not fully known until the EMD build. 

Should foster competition in production. 

TDP should be proven in production by another source. 

These comments express a concern with selection of a production contractor based 
on cost estimates generated in Dem/Val and the opinion that it is in the Government’s 
best interest to have competition in the first production. 

Overall, the techniques with the greatest differences in usage rates between Natick 
and non-Natick respondents were due to disagreements about the value of the techniques 
and their inherent risk levels, as well as commodity and command structure related 
differences. 

Highest Usage Comparison - To analyze Table 1, it was decided to 
comparatively analyze the highest usage rate techniques for the Natick and non-Natick 
groups. Tables 4 and 5 outline the streamlining techniques that received the highest 
response rates. The ten techniques with the highest Natick usage and the eleven 
techniques (there was no distinction between usage rates of the tenth and eleventh 
highest) with the highest non-Natick usage are presented for comparison, to investigate 
similarities, differences and the reasons for their high usage. 

Common Techniques - As depicted in Tables 4 and 5, of the high ten or eleven 
techniques of the two groups, six are common to both groups. These will be analyzed 
individually for the reasons for their high usage and commonality. 

The reasons for the high usage of Technique 1 (Combine Milestones 0 and I) 
were the low risk and high technological maturity of AC AT IV programs. Specific 
comments included "Items always of low technical risk and do not warrant concept 
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ACAT IV Streamlining Techniques 

Frequency of Use by Command, Highest Natick and Non-Natick Responses 

Shading = Common Techniques 



L Combine Milestones 0 and I {Skip CE 
Phase) 


2, Combine Milestones I and II (Skip 
Detn/Vat Phase) 


4* ^C^mbine! nem/V^J •* Uev elopment and 
Operational Toting . 


11. Request Milestone Decision Authority 
delegate Milestone Decision Review 
Chairmanship to RDEC Management 


i3;;1staff Jpestone Decision Package prior 
to receipt of Independent 
,lv|iijadatilAss^s&int ippdrts 


.t4f iJ^’inmdin^A^e&adeiMl: di Evaluation 
:a^sidnf (Deasibn.: 


16. Waive requirement for a Cost and 
Operational Effectiveness Analysis (COEA) 


18. Waive requirement for a separate 
Logistics Demonstration 


24. Utilize a single contract for Dem/Val 
and EMD 


25. Utilize a single contract for EMD and 
First Production 



All Other Commands 


I,- Combine Milestones 0 and I (Skip CE 
Phase) |:>p- l;£fP; 


2.. Combine Milestones I and H {Skip } 0 : 
Dem/Val Phase) 


4. Combine Dem/Val Development and 
Operational Testing ||I! 


5. Conduct concurrent Dem/Val 
Development and Operational Testing 


7. Conduct concurrent (overlapping) EMD 
DT and IOT&E 


13.' Staff. Milestone Decision Package prior , 
to receipt of Independent 
’ Evaluation/Assessment;::!^^ 


• 14. Use Interim Assessment dr : Evaluation , 
Reports to support a Milestone Decision J- 


17. Waive requirement for a separate 
System Threat Assessment Report (STAR) 


24. Utilize a single.contract. for Dem/Val 
and EMD 


26. Conduct production First Article Tests 
to validate development deficiency 
corrections 


27. Validate technical manuals/support 
system during first production instead of 
during EMD 


Table 4. Frequency of Use by Command 


























ACAT IV Streamlining Techniques 

Frequency of Use by Command 

Highest Natick and Other Commands' Responses 


Command 


No. of 

Respondents 


Streamlining 

Technique 


Natick 


Other Commands 
Total 



# Yes 


% Yes 


# Yes 


% Yes 



Shading = Highest Usage for Each Group 


Table 5. Frequency of Use, Highest Responses 




















exploration" and "Small programs usually don’t need to prove out technology to the 
extent of more complex ones". Systemic problems with the development process were 
noted as driving the decision to combine the Milestones as well. It was noted that 
Milestone 0 cannot be conducted without an approved Mission Needs Statement (MNS) 
and that pressures to develop items quickly necessitate initiating development prior to the 
receipt of the approved MNS. The idea is to obtain sufficient progress by the time the 
MNS is approved to conduct the Milestone I Decision Review. In addition, in many low 
tech/low risk programs, the activities required for Milestones 0 and I are almost 
synonymous. 

The reasons cited for the high usage of Technique 2 (Combine Milestones I and 
II) were also risk related. The use of a single build/test phase fits the needs of most 
AC AT IV programs, because they are frequently modified NDI efforts. A key factor 
noted in the selection of this technique was the degree of success in the Concept 
Exploration Phase and the demonstration of the required technologies. The technique is 
used because it saves a great deal of funding and program time. Respondents from one 
Command, TARDEC, disagreed with its use citing that there was no reason to assume 
the additional risks when, in their view, the test costs were low. The implication was that 
Dem/Val is necessary to avoid costly problems in EMD. This difference may be due to 
commodity differences, or differing degrees of risk acceptance. 

Technique 4 (Combine Dem/Val DT and OT) was frequently used because ACAT 
IV programs are relatively simple and have few operational issues. The respondents felt 
that DT with user troops was sufficient to identify and verify user issues. However, even 
though the technique had high usage, there were difficulties associated with 
implementation. Specific concerns were "Often difficult to get OEC and TEXCOM to 
agree" and "Ability to get concurrence from DT and OT testers/evaluators". An 
additional programmatic concern noted was that Operational Requirements Document 
(ORD) approval is required approximately one year prior to the start of OT. This could 
create a situation where DT is held up, awaiting the ORD, if a combined test were 
planned. 
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Techniques 13 (Staff Milestone Decision Package prior to receipt of Independent 
Evaluation/Assessment Reports) and 14 (Use Interim Assessment or Evaluation Reports 
to support a Milestone Decision) address the same fundamental issue and the reasons for 
their high usage are the same. The issue is that programs approaching a Milestone 
Decision have time-lines that require specific Review support documentation, and the 
conduct of the Review often falls in a shorter timeframe than the evaluators/assessors can 
support with fully staffed and coordinated formal reports. Reasons cited by the 
respondents keyed on the need to meet the Milestone date and award follow-on contracts. 
They emphasized that use of these techniques presented little risk, since the "results are 
usually known long before the final report". It was interesting to note that the narrative 
comments keyed on the time saved and low risk, but did not indicate that the respondents 
felt that the evaluators/assessors took too long. This is surprising since the evaluations 
are usually on the critical path leading up to the Decision Review, and the relationship 
between the development managers and evaluators is typically adversarial. 

Technique 24 (Utilize a single contract for Dem/Val and EMD) was cited as a 
very large saver of funding and time, while actually lowering the risks associated with 
two contracts (award problems, transfer of knowledge/technologies, etc.). One 
respondent noted: 

Regarding ACATIV programs, the time to complete Dem/Val may be the 
time to award a contract. Purpose of separating is to bring a complex 
effort into manageable pieces thereby reducing risk. In ACAT IV we 
usually have much less complexity and are able to manage the Dem/Val 
and EMD with less risk and greater ease. 

This appears to be an almost universally accepted strategy when the program 
includes both the Dem/Val and EMD phases. However, difficulties arise when trying to 
write a sufficiently detailed Scope of Work early in Dem/Val, given that specific 
requirements may not yet be fully defined. 
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Unique Techniques - In the top ten/eleven usage techniques, there were four that 
were unique to Natick and five that were unique to the other Commands. Each of these 
will be evaluated for reasons for the differences. 

Natick Unique Techniques - Technique 11 (request MDA delegate Milestone 
Decision Review Chairmanship to RDEC management) was unique to Natick. This 
technique was analyzed earlier as having one of the largest usage differences between 
Natick and non-Natick respondents. Reasons for the differences are cited previously. 

Technique 16 (Waive requirement for a COEA) was unique to Natick, but the 
narrative comments from the other Commands also supported the technique. The COEA 
was generally considered too costly and inaccurate. The only difference was that a 
number of the non-Natick respondents felt that the decision was up to TRADOC, and 
they were not authorized to decide. Therefore, they did not respond positively. 

Technique 18 (Waive requirement for a separate Logistics Demonstration) was 
unique to Natick. According to the narrative comments, for non-Natick commodities, the 
demonstration is not required in many cases (for ammunition for instance). It appears to 
be a commodity specific requirement at the ACAT IV level, dictated by the directions 
from AMSAA. 

Technique 25 (Utilize a single contract for EMD and First Production) was unique 
to Natick. As cited previously, the non-Natick respondents expressed concern with 
selection of a production contractor based on cost estimates generated in Dem/Val and 
they considered it in the Government’s best interest to have competition in the first 
production. The reasons for the difference may be due to commodity differences relating 
to item complexity and contractor past performance, and/or differences in organizations’ 
affinity for risk. 

Non-Natick Unique Techniques - Technique 5 (Conduct concurrent Dem/Val DT 
and OT) was unique to the non-Natick respondents. This is interesting since Technique 
4 (Combine Dem/Val DT and OT) was also one of their highest usage techniques and 
these two techniques are mutually exclusive. A number of non-Natick respondents 
commented on both techniques with the same statements. This implies that they either 
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did not reco gniz e the difference or they would consider both and select the best one for 
the situation at hand. In addition those respondents who stated that Technique 4 was too 
risky agreed with Technique 5. One who disagreed with the use of Technique 4 stated 
regarding Technique 5 that he would "Definitely support this, DT and OT (are) not 
necessarily dependent on each other. Concurrent testing will bring you to completion 
quicker, shorten project schedule and labor dollars". However, Natick respondents 
clearly preferred Technique 4 over Technique 5. They indicated a preference for 
combined testing over concurrent testing in Dem/Val. Non-Natick respondents cited risk 
most often as the factor in their decision to select concurrency. Natick managers appear 
to be more willing to accept greater risk in the area of test phasing. This difference in 
philosophy may be due to commodity differences or past experiences on the part of either 
group. 

Technique 7 (Conduct concurrent EMD DT and OT) was unique to non-Natick 
respondents and this mirrors the results regarding Technique 5 above. Natick respondents 
preferred combined testing (Techniques 4 and 6) and non-Natick respondents preferred 
concurrent testing (Techniques 5 and 7). Both groups supported acceleration of testing 
in both Dem/Val and EMD. But differences in opinion concerning acceptable risk levels 
leads to a clear difference of opinion on how to accomplish this acceleration. 

Technique 17 (Waive requirement for a separate STAR) was unique to the eleven 
highest usage techniques of the non-Natick group. However, the Natick usage level was 
also very high, although not in their top ten. In addition, it is clear from the narrative 
comments that there is general agreement that the technique is supported and that a 
STAR for ACAT IV programs is generally not necessary and not required of the 
managers. 

Technique 26 (Conduct production First Article Tests to validate development 
deficiency corrections) was unique to the eleven techniques used most often by the non- 
Natick group. However, the Natick usage level was also very high, although not in their 
top ten. In addition, it is clear from the narrative comments that there is general 
agreement that the technique is supported. Validating development deficiency corrections 
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has become one of the purposes of First Article Testing (FAT). It makes good business 
sense to validate the corrections in First Article instead of conducting a separate 
validation test prior to a Milestone Decision and FAT. Combining of testing needs is 
being applied here and as noted earlier with respect to DT and OT in Dem/Val and 
EMD. 

Technique 27 (Validate technical manuals/support system during first production 
instead of during EMD) was unique to non-Natick respondents. The non-Natick 
respondents indicated in their narrative comments that this would be a better test of the 
support system and it avoids unnecessary changes to the manuals/support system due to 
configuration changes in EMD. This technique was not used by Natick, as respondents 
indicated that waiting until production was too risky, and that the manuals and support 
system were required for evaluation during IOT&E. This may be a result of past attempts 
by Natick managers to institute this technique that were not supported by other 
acquisition team members. Commodity differences may have allowed other Commands 
to get acquisition team approval for this technique when Natick had not been able to. 

Summary - In summary, the high usage streamlining techniques used by AC AT 
IV managers that are common to Natick and the other AMC Commands can be 
categorized into two groups, major program events and programmatic 
reporting/documentation. 

The common major event techniques address combining, or concurrency of 
events. Managers combine Milestones 0 and I, or I and II; they conduct combined 
(Natick) or conduct concurrent (non-Natick) Dem/Val and EMD DT and OT; they use 
a single contractor for Dem/Val and EMD; and they utilize FAT to validate development 
deficiency corrections. The techniques are chosen because they save large amounts of 
time and funding, and they have an acceptable level of risk for AC AT IV programs. The 
common programmatic reporting/documentation techniques are the acceleration of 
independent evaluator/assessor reporting for Milestone Decision Reviews, and the 
waiving of the need for a STAR, or COEA for their programs. 
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The high usage techniques that were unique to either the Natick or non-Natick 
group appear to be due to risk acceptability differences that may be due to commodity 
differences or past experiences. It was noted that Natick managers were consistently 
greater risk-takers regarding test phase streamlining. 

B. TECHNIQUE-BY-TECHNIQUE COST SAVINGS AND SCHEDULE 
REDUCTION COMPARISONS 

To evaluate the measures of effectiveness associated with the cost and schedule 
savings, this section will begin with an overall comparison of the cost savings of small 
and large programs. Then the techniques with the highest savings will be evaluated and 
compared with the techniques established in the last section as having the highest usage. 
Comparisons will then be made between small and large programs regarding the 
differences in high saving techniques. Finally, the techniques with the lowest savings will 
be evaluated, as opposed to the techniques with the highest usage, and then by comparing 
small versus large programs. As noted earlier, the lowest saving techniques are being 
evaluated because a number of them were both low in savings and high in use, which has 
an interesting basis. 

1. Overall Comparison 

Before evaluating the highest and lowest saving techniques, a review of all 
techniques by program sizes revealed an interesting relationship. The relative technique- 
to-technique average cost savings established by the respondents from small and large 
programs, Figure 5, is very consistent. Cost savings for small programs are equal to, or 
slightly lower than, large programs for almost every technique. A similar comparison of 
the schedule reduction ratings for small versus large programs showed no clear trend. 
Note that the numbers of observations for schedule reduction in the small program 
category were small, with a number of cells containing very few or no responses (see 
Table 2). 
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Cost Savings 
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Figure 5. Small vs. Large Program Cost Savings 





The cost saving comparison indicates a general agreement between small and large 
program managers as to the relative cost saving potential of each technique. There was 
general agreement on which techniques saved the most and which saved the least. The 
four techniques with the largest separations between small and large programs will be 
analyzed individually for reasons for the large differences. 

Technique 6 (Combine EMD DT and IOT&E) saved significantly more for large 
programs. The reason is not apparent from the narrative comments of the respondents. 
However, it may be due to the increased complexity of larger systems and the associated 
greater test costs. 

For Technique 12 (Request DA-Directed Procurement), the small program 
response rate (and associated projected cost savings) was zero. Since use of this 
technique is urgency driven (and then only as a last resort), small programs may never 
have the level of urgency or importance to warrant its use. This is supported by 
comments from small program managers that included not seeing any occasion for using 
the technique. 

Technique 17 (Waive requirement for a separate STAR) was consistently indicated 
by small program managers, as well as many large program managers, as not being 
required. Therefore, the benefit to them was small. A few large program managers 
indicated savings and some of their programs may be threat driven and warrant 
significant effort with respect to threat assessment. One large program manager noted he 
supported the waiver concept "if the system will be in a limited or no combat 
environment". 

Technique 24 (Utilize a single contract for Dem/Val and EMD) had significantly 
higher cost savings for large programs. Although the narrative comments did not provide 
a reason for the difference, it may indicate that contract preparation, evaluation, and 
execution cost savings for larger programs are greater due to increased system 
complexity. As the complexity of a system increases the development costs increase, and 
higher cost development contracts involve increased regulatory requirements and more 
rigorous proposal evaluation procedures, both of which increase administrative costs. In 
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addition, consolidation of two complex contract requirements can save substantially by 
eliminating the second contractor’s start-up costs. Also, a contractor for the combined 
effort can propose a lower cost program when a longer duration effort is guaranteed. 
These points indicate that as a program size increases, the potential savings from 
consolidation of Dem/Val and EMD contracts increases as well. 

Overall, there is general agreement between small and large programs regarding 
the relative cost saving potential of each of the streamlining techniques. Two areas where 
large programs deviate from the trend are combining EMD DT and IOT&E, and using 
a single contract for Dem/Val and EMD, both of which may be due to the increased 
complexity of larger programs. Additionally, large programs have a greater need for, and 
larger cost savings from, threat and urgency related techniques. 

2. Highest Saving Techniques 

The techniques with the highest cost savings and schedule reduction are 
highlighted in Table 6. The top six techniques in each category were selected for 
evaluation. By comparing the techniques with the highest savings to those with the 
highest usage, as well as the differences between small and large programs, Figure 6, 
a number of interesting similarities and differences can be noted. A comparative analysis 
of the techniques with high savings in at least two categories and those with the highest 
usage yields five techniques in both categories and three that have high savings but were 
not highly utilized. The following discussion first considers techniques with both high 
savings and high usage, then it addresses those with high savings and low usage, and 
finally it examines the differences between small and large programs. 

High Savings and High Usage - Techniques 1 (Combine Milestones 0 and I), 2 
(Combine Milestones I and II), and 4 (Combine Dem/Val DT and OT), from the above 
high usage discussion were noted by the respondents as having been selected because 
they save large amounts of time and funding. Specific comments were: 


51 




Projected Savings by Program Size 

Highlighting Highest Savings Shading = Highest Usage and Ratings 





Table 6. Techniques with the Highest Savings, Shading = Highest Usage and Highest Ratings 
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Figure 6. Projected Savings by Program Size, Techniques with the Highest Savings. 










































Technique 1 - The total manpower costs and schedule associated with 
doing a Milestone 0 will be somewhat costly, today we are told that we 
need to deliver products in a few years, doing an additional step doesn’t 
streamline the process. 

Technique 2 - Shortens time and reduces number of documents. 

Technique 4 - Cost of testing is very expensive, schedule can be greatly 
shortened, in some situations the acquisition cycle can be cut in half. 

The managers agree that, for low risk programs, skipping phases and combining 
tests makes sense. For Technique 1 there is a lack of a need for the event and for the 
other two techniques the increase in risk is not unreasonable for low complexity, 
technologically mature programs. 

Technique 18 (Waive requirement for a separate Logistics Demonstration), a high 
Natick usage technique, also had a high payoff for small programs in terms of both cost 
and schedule savings. The need for a logistics demonstration was stated by one 
respondent as "depending on the complexity of the system". Therefore, smaller programs 
would have less of a need for a separate event, and imposing a separate event would have 
a greater impact on their programs in terms of percentage of the total cost and schedule. 

Technique 26 (Conduct production First Article Tests to validate development 
deficiencies), as stated in the above high usage analysis, follows the overall philosophy 
of combining events, especially tests, because combining saves the most in terms of 
program funding and schedule. Managers are fulfilling the need to validate the 
corrections without having to run a separate event. It makes good business sense and is 
accepted as "business-as-usual" by AC AT IV managers. 

Although the numbers are not statistically significant, for the above high savings 
and high usage techniques the ranges of cost and schedule savings noted by the 
respondents were as follows: 
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Small Programs 

Cost Savings 

Schedule Reduction 

Technique 1. 

N/A 

1 - 18 MOS. 

Technique 2. 

$0- >$100K 

1 - 18 MOS. 

Technique 4. 

$100- >$100K 

N/A 

Technique 18. 

$0- >$100K 

1 - 18 MOS. 

Technique 26. 

$10K - $100K 

1 - 18 MOS. 

Large Programs 

Technique 1. 

$10K - >$100K 

1 - >18 MOS. 

Technique 2. 

>$100K 

1 - > 18 MOS. 

Technique 4. 

$10K - >$100K 

N/A 

Technique 26. 

N/A 

1 - > 18 MOS. 

These ranges of savings 

are generally too broad 

to be of value to ACAT IV 


managers in their streamlining decision-making. However, the large program cost savings 
ranges for Techniques 1 and 2 are of value. The quantity of respondents answering each 
was nine and their responses were very consistent. 

High Savings without High Usage - Technique 3 (Combine Milestones I and III) 
presents the potential for the greatest savings since it eliminates most of the development 
process. But its applicability is limited to NDI and modified NDI programs because the 
technique allows for very little test and evaluation. As stated by one respondent "Not 
applicable to all systems, combining of the two for an NDI makes sense". 

Technique 6 (Combine EMD DT and IOT&E) presents the potential for savings 
of the same order as Technique 4, which had high savings and high usage. But the 
respondents felt that a combined test in EMD was much more difficult and risky than one 
in Dem/Val. Comments included "Risk of failure is high unless prior effort was very 
successful", "Often difficult to get OEC and TEXCOM to agree", and "Too hard to 
schedule user troops". 
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Technique 10 (Conduct user evaluation in lieu of formal IOT&E) was noted as 
a large saver of funding but not time. Comments were that IOT&E was expensive, but 
that the evaluators and assessors would not accept the results of an informal user 
evaluation in lieu of IOT&E. The technique could save significant funding but the lack 
of tester control and data collection capabilities make the informal evaluation 
unacceptable in most cases. 

In summary, the high use of the combining of milestones and testing phases by 
ACAT IV managers is supported by the high savings in program costs and time that is 
associated with these streamlining techniques. For small programs the Logistics 
Demonstration is less useful and elimination represents a significant percentage of the 
program’s cost and schedule. Other techniques that could save significant amounts of 
funding and time are unique to special cases such as NDI programs or are impractical 
and/or risky. 

Small versus Large Programs Comparison - With respect to cost savings, four 
techniques are common to small and large programs. Techniques 2 and 3, which combine 
Milestones I and II, and I and III, respectively, were high cost savers for both size 
programs. This is understandable given the comments previously noted, that these 
techniques save significant amounts of time. Technique 1 (Combining of Milestones 0 
and I) was only considered a significant cost saver for large programs. For small 
programs the concept exploration phase did not involve significant funding. For low 
technology systems that require no concept testing, but only require a concept prove-out 
paper study, this result can be expected. 

In addition. Technique 4 (Combining of Dem/Val DT and OT) was common to 
both groups. This was noted previously as a very high saver. Finally, Technique 10 
(Conduct user evaluation in lieu of IOT&E) was common to both size programs. The 
respondents comments noted that formal OT, especially IOT&E, was very expensive. 

High cost saving techniques that were unique to either small or large programs 
were Techniques 6, 18, and 26. Technique 6 (Combining of EMD DT and IOT&E) was 
unique to large programs and the reason was not discemable from the respondent’s 
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comments. The difference may be complexity related, where the cost of IOT&E increases 
substantially as program complexity increases. Technique 18 (Waive requirement for a 
separate Logistics Demonstration) was unique to small programs. As discussed 
previously, this technique saves a significant percentage of a small program’s funding and 
time, where low complexity warrants its use. Technique 26 (Conduct production FAT 
to validate development deficiency corrections) was unique to small programs but only 
two respondents contributed to the high score. 

Overall, the same techniques, the combining of Milestones and test phases, save 
the most funding for both small and large programs. The one exception was that the 
combining of Milestones 0 and I for small programs did not save significant dollars, 
possibly because the proof of concept for low technology/complexity systems requires 
little, if any, breadboard testing and usually only requires a paper study. 

With respect to schedule reduction, the results were very similar to the cost 
saving results. The combining of Milestones <0/1,1/II, and I/III) were all high schedule 
savers for both small and large programs. The combining of any Milestones, even 
Milestones 0 and I for small programs, saved significant amounts of program time. The 
combining of EMD DT and IOT&E, Technique 6, was also a common high schedule 
reducer. But the combining of Dem/Val DT and OT, Technique 5, was not, even though 
it was a high cost saver for both groups. As was noted in the previous section, the 
complexity of testing in EMD and the length of formal IOT&E may contribute to the 
difference. 

Additionally, Technique 26, using FAT to validate development deficiency 
corrections, was common to both groups. By utilizing the planned FAT to validate the 
deficiency corrections, the program can avoid a separate event and can proceed out of 
development, both significant schedule impacts. 

The one technique that was unique to small programs was Technique 18 (Waive 
requirement for a separate Logistics Demonstration). See the previous discussions 
concerning analysis of this point. 
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Highest Saving Techniques Summary - Overall, the greatest time and funding 
savings can be obtained by combining Milestones and test phases. Concurrency of testing 
was not one of the highest savers, even though it was noted as a high usage technique, 
particularly for Commands other than Natick. Small and large programs are very similar 
with respect to the highest saving techniques, with a few exceptions. For large programs 
the expected range of cost savings associated with Techniques 1 (Combine Milestones 0 
and I) and 2 (Combine Milestones I and II) are greater than $10K and greater than 
$100K, respectively. 

3. Lowest Saving Techniques 

The techniques with the lowest cost savings and schedule reductions are 
highlighted in Table 7. As discussed previously, the lowest saving techniques are being 
evaluated because a number of them were both low in savings and high in use. The 
lowest six techniques in each category were selected for evaluation. The amounts of 
funding saved are too small to be of concern in the decision-making process. However, 
the amount of time saved might be of concern, if the program were in a time critical 
phase (such as leading up to a Milestone Review or contract award). By comparing the 
techniques with the lowest savings to those with the highest usage, and the differences 
between small and large programs, Figure 7, a number of interesting similarities and 
differences are noted. 

Low Savings and High Usage - A comparative analysis of the techniques with 
the lowest savings in at least two categories and those with the highest usage yields five 
techniques in both categories and three that have low savings and that were not highly 
utilized. 

Techniques 13 and 14 are the two high usage techniques concerned with the 
acceleration of Independent Evaluation/Assessments for Milestone Decision Reviews. As 
discussed earlier, the reason for the high usage was due to the timing of the Milestone 
Decision Review. The time-line for a Review is very tight and slippage is usually 
considered unacceptable, especially if follow-on contract award (and obligation of 
unprotected funds) is awaiting the decision. With events measured in days and weeks it 
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Projected Savings by Program Size 

Highlighting Lowest Savings Shading = Highest Usage and Lowest Ratings 
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Table 7. Techniques with the Lowest Savings, Shading = Highest Usage and Lowest Ratings 







Projected Savings by Program Size - Techniques with Lowest Savings 



Figure 7. Projected Savings by Program Size, Techniques with the Lowest Savings 


























































is understandable why the small savings associated with the use of these techniques 
warrants their broad use. The high use may indicate that these techniques become 
necessary due to slippages leading up to the Review, or delays by the 
evaluators/assessors. But the respondent comments indicate that they are utilized because 
they make sense, they save time at a critical point in the process and there is little risk 
with their implementation. 

Technique 27 (Validate technical manuals/support system during first production 
instead of during EMD) also saved little, in terms of schedule, but it was highly utilized 
by Commands other than Natick. The cost savings associated with this technique were 
not high, eliminating cost as the reason for the high usage. The non-Natick respondents 
indicated that this technique was a better test of the support system, whereas Natick 
respondents saw it as risky, or unrealistic (has to be done prior to IOT&E). The non- 
Natick respondents appear to favor the technique not because of cost or schedule savings 
considerations, but because it provides a higher quality product, a factor outside of the 
considerations of the survey. 

Techniques 16 (Waive COEA) and 17 (Waive STAR) were also highly utilized 
with low savings. As noted earlier, these techniques appear to be a standard way of doing 
business for most AC AT IV programs. The documents are generally not required and 
they should be considered for generation on a "by exception" basis only. 

Low Savings without High Usage - Techniques that had low savings and were 
not highly utilized indicate the difficulty in their utilization and their lack of usefulness. 
Technique 12 (Request DA-Directed procurement prior to Type Classification-Standard), 
as discussed earlier is difficult to get approved and is inappropriate except for in extreme 
emergencies. Technique 15 (Use Independent Evaluator Briefing in lieu of written reports 
to support a Milestone Decision) is shown to be a less desirable option to Techniques 13 
and 14, and it accomplishes the same thing (accelerate the evaluation process). 
Respondents preferred some form of written report in advance of the Decision Review. 
Technique 19 (Utilize TIWG as E3 Requirements Board) doesn’t appear to save program 
time or funding. It merely saves the need for two staffing and coordination groups. 
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Small versus Large Programs Comparison - With respect to cost savings, all 
of the lowest saving techniques are common to both small and large programs. As noted 
earlier, the amounts of funding saved are relatively meaningless and the narrative 
comments contain no basis for this agreement. 

With respect to schedule reduction, Techniques 15 and 19 are common to small 
and large programs, and as noted earlier they are not highly utilized. They uniformly 
provide little savings for the reasons noted above. Technique 27 is also common to both 
groups and, as noted earlier, is utilized by non-Natick respondents to provide a better 
product. Techniques 13 and 14 are unique to large programs with respect to their low 
schedule savings. The ratings for small programs are also fairly low, with averages that 
are actually lower than the large program amounts. Note, also, that the small program 
number of observations are very low in some cases. 

Lowest Saving Techniques Summary - In general, small schedule reductions are 
important when they occur close to key events, such as the Milestone Review. Waivers 
of documentation, such as the COEA and STAR, do not create significant savings, but 
they are generally not applicable to, nor required of, ACAT IV programs. Some 
techniques, such as validation of the support system in production, are valuable to some 
respondents because of improved quality, not savings of time or funding. Some 
techniques are of little to no value because they either do not save enough to matter, or 
because better alternative streamlining methods exist. There are no important differences 
in savings between small and large programs that are discemable from the data. 

C. COMMANDS’ ACAT IV MANAGEMENT COMPARISON 

As noted earlier when excluding MICOM and CECOM from the analysis, the 
management of ACAT IV level programs in their commodity areas differs from that in 
the areas supported by the other AMC Commands. This difference will be investigated 
by first describing the differences between the two management structures, then exploring 
the reasons for the differences, and finally evaluating the pros and cons of each. The 
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Appendix A. POCs’ comments and the researcher’s personal experience as a manager 
at Natick form the basis for these comparisons. 

The management of small development programs within the U.S. Army generally 
travels along the following path: (1) the AMC RDECs and Army Research Laboratory 
(ARL) investigate technologies and consider concepts for development (Technology Base 
and Concept Exploration Phases); (2) for the development phases, some programs 
transition to the Program Executive Officer (PEO) management structure, others remain 
with AMC and are managed by AMC level PMs, and still others are left with the AMC 
RDECs; (3) following development and fielding, management transitions to the AMC 
MSCs for sustainment and eventual disposal. In the cases of CECOM and MICOM, none 
of the programs remain in AMC when they transition into development, they all 
transition to PEO management. 

Both of these management philosophies appear to be allowed by DoD 5000, but 
the method employed for the CECOM and MICOM commodities does not appear to meet 
the intent of DoD 5000 or its basis, the Packard Commission Findings, both of which 
were discussed in Chapter II. The required Milestone Decision Authority levels for each 
of the ACATs in DoD 5000 are listed in Figure 2. In addition, the program management 
levels associated with each AC AT level are defined in the Army’s implementation 


regulation for DoD 5000 (AR 70-1) and the criteria for their selection are listed below: 

ACATs 

Program Management 

Primary Criteria 

I 

PEO PM 

$300M RDTE, $1.8B Procurement 

II 

PEO PM 

$115M RDTE, $540M Procurement 

III 

PEO or AMC PM 

High Visibility, Special Interest 

IV 

Systems Manager 

All Other Acquisition Programs 


From these criteria, it can be established that only the more important lower cost 
programs should require certified PMs. In addition, the mission of the PEO is stated as 
the cost-effective acquisition of major new weapon/support systems. Whereas, the 
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related AMC missions are the acquisition of non-PEO new weapon/support systems and 
the imp rovement and sustainment of mature, fielded systems. Although there does not 
appear to be a restriction against PEO management of all development programs in a 
commodity area, from the above statements the intent of the DoD and the Army appears 
to be that non-certified AMC managers should manage the development of the smaller 
new systems in each of the commodity areas. 

There may be some commodity related explanations for this difference. Programs 
remain with Natick and ERDEC because there are no PEOs controlling their 
commodities. TARDEC encompasses ARDEC munitions and support items that have 
carried over from the disestablished Belvoir RDEC that do not have related PEOs. In 
addition, AATD manages a number of aviation support items that are not controlled by 
the PEO Aviation. In the cases of CECOM and MICOM, the AMC Commands and the 
PEOs are directly aligned, with respect to commodities, and they are co-located. Of the 
above three Commands, the only one with a situation similar to CECOM and MICOM 
is AATD. This alignment and the PEOs’ degree of control over funding decisions may 
have led to a clear distinction between the missions of the two, with an evolution to PEO 
control of all development. 

The disadvantage of the MICOM/CECOM model, according to the MICOM 
POC, is that the PEOs have assumed control of all development and upgrade 
responsibilities, and AMC has become completely focused on supporting the PEO, 
relinquishing their mission to improve mature, fielded systems. The concern of MICOM 
is that the PEOs do not fully consider system upgrades when evaluating the need for a 
new or improved capability. They concentrate on developing new systems. 

The above concerns could be labeled as an AMC Command complaining about 
the PEOs and their own loss of control over programs and funding. With the 
establishment of the PEO structure in the late 1980s, the AMC Commands such as 
CECOM and MICOM lost a great deal of funds control and power. In addition, much 
of AMCs technical efforts transitioned to a role directly support the PEOs, and the PEOs 
control of funding gives them control over AMCs staff. Both of these factors have 
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sometimes lead to a power struggles and attempts at "mission-poaching". However, the 
fact that the PEOs control all developments indicates a degree of over-management that 
was not intended by the Packard Commission nor expressed in DoD 5000, where ACAT 
III and IV programs do not require PEO management. The Packard Commission findings 
created the framework for the Acquisition Executive and PEO structure and they were 
specifically directed at major weapon systems.[Ref. 25] DoD 5000 allows for ACAT III 
and IV management at the lowest level deemed appropriate by the Acquisition Executive 
(see Figure 2). There is clearly a place for management of ACAT IV size programs by 
individuals who are not certified PMs, as noted by the ACAT structure in DoD 5000 and 
the allowance for AMC management of ACAT III and IV programs. In fact, this 
management structure is in place at the other AMC Commands that participated in this 
study. 

On the other hand, AMC management of existing programs and funding lines 
suffers from a lack of visibility and power when competing with the PEOs. The PEOs, 
if they fully considered upgrades when evaluating a new requirement, and if they 
managed small programs without creating new certified PMs, would be more efficient 
as the single point of contact for all developments in a given commodity area. In 
addition, even the smallest programs would benefit from the green-suit management 
available within the PEOs. 

D. OTHER COMMENTS 

Two streamlining techniques that were proposed by the respondents to the survey 
are worth mentioning, at least for future reference or use in follow-on studies, 

1. Use of capstone Program Management Documentation such as TEMPs, ORDs. 

Allows development of several related efforts without waiting for separate staffing 

and creation of multiple documents. 

2. Use of multi-year production contracting can result in long-term cost and time 

savings. 
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The first of these techniques recommends that organizations, rather than individual 
AC AT IV managers, develop and maintain documentation that is required for the 
development of new systems within their commodity areas. All systems requiring 
development in a commodity area such as heavy cargo and resupply airdrop or a family 
of field heaters, for instance, could be supported with commodity level requirements 
documents (Acquisition Strategies, Test and Evaluation Master Plans, etc). For low 
complexity systems (ACAT IV), over-arching documents may unburden individual 
managers and, overall, save funding. Coordination of changes to the documents as the 
various efforts evolve, however, could be a burden and the process could create its own 
bureaucracy. But the concept is certainly worth considering for some systems and 
commodities. 

The second technique addresses the production phase of materiel acquisition which 
this study considered only when a streamlining technique involved a transition from 
development into production. If the goal of development, and the acquisition process in 
general, is fielding of new and improved systems, production is as important as 
development. When considering methods for cutting the cost and time required to field 
new systems, production should be considered if the entire process is to be improved. 
However, for many small systems, the management of the program transitions from the 
RDEC to a procuring command prior to these decision. Analysis of both phases together 
would require investigating two distinctly different set of issues with two completely 
different groups. Therefore, it is suggested that both groups be investigated, but in 
separate, parallel efforts. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


The Primary Research Question for the study was: "What acquisition streamlining 
techniques are utilized by U.S. Army ACAT IV managers?" By conducting a survey of 
all of the Army’s ACAT IV development managers, from all of the Commands doing 
ACAT IV level development, this thesis sought to ascertain the differences among 
Commands and determine which techniques are most prevalent, most useful, and the 
circumstances that cause them to be successful. 

In support of this thesis, 39 questionnaires were distributed to the ACAT IV 
development managers at various U.S. Army AMC Commands. Two Commands, 
CECOM and MICOM, were eliminated from the survey as is was determined that all of 
their development programs were at the ACAT III level or higher. A total of 19 
responses were obtained, with some Commands providing limited responses. Given this 
response level, a limitation was created as it was difficult to establish statistical 
significance for the results, since the quantity of observations was lower than expected. 
However, the data that was received was deemed to be valid. Therefore, the analysis was 
conducted on a qualitative, rather than predominately quantitative, basis. Following are 
the conclusions and recommendations drawn from this analysis, as well as 
recommendations for further research in the subject area of acquisition streamlining of 
small DoD development programs. 

A. CONCLUSIONS 

Subsidiary Research Question 1. - How do Army Commands streamline 
ACAT IV programs differently? The purpose of this segment of the study was to 
determine if all U.S. Army Commands were aware of all available streamlining 
techniques and establish if there were differences in the Commands usage levels or 
patterns. 

The first area to be analyzed was Command awareness of the techniques. The 
investigation found that of the 25 techniques used among the five Commands, there was 
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only one instance where none of the respondents from a particular Command where 
aware of a specific technique. Based on this, it can be concluded that cross-fertilization 
of the identified streamlining concepts exists among the U.S. Army Commands. The 
information is available to all of them. 

The second area of analysis was a comparison of the techniques with the greatest 
differences in usage levels among the Commands. The relative usage levels for the 
various techniques is consistent among all Commands. For those techniques where the 
usage differed the most, the reasons were disagreement with the value or risk of the 
technique, or commodity/Command differences. The specific techniques with the greatest 
differences were: 

1. Combine Dem/Val DT and OT 

2. Utilize a single contract for EMD and First Production 

3. Waive requirement for a separate Logistics Demonstration 

4. Validate technical manuals/support system during first production instead of 

during EMD. 

5. Request MDA delegate Milestone Decision Review Chairmanship to RDEC 

management 

The first two techniques were utilized more often by Natick managers because the 
managers considered their level of risk acceptable. For the third technique, managers in 
the Natick commodities areas were more likely to have this requirement imposed upon 
them. Therefore, the need for them to consider a waiver was necessary more often. The 
fourth technique was more likely to be used by managers at Commands other than 
Natick. The managers at these other Commands saw this technique as providing a higher 
quality product. However, difficulties in getting acquisition team consensus and the 
timing of events made it less attractive to Natick managers. The need for the fifth 
technique was based on Command structure differences, where the more remotely located 
managers favored having the Chairmanship delegated. 

The final area of analysis for this sub-question addressed the techniques that were 
utilized most often by all of the Commands. A number of streamlining techniques are 
used routinely by most ACAT IV programs. For techniques affecting major program 
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phases and events, the risk levels associated with actions such as the combining of 
Milestones and the concurrency of test phases are considered acceptable for programs at 
this level of complexity. Although quantifiable ranges for the levels of savings expected 
for the techniques could not be concluded from the research (because of the low number 
of participants and respondents) the following can be concluded as the key techniques 
utilized by most AC AT IV managers. 

1. Combining of Milestones 0 and I, and/or I and II 

2. Combining of test phases, both in Dem/Val and EMD, for Natick managers 

3. Concurrency of test phases, both in Dem/Val and EMD, for non-Natick 

managers 

4. Combining development deficiency correction validation testing with First 

Article Testing 

5. Utilize a single contract for Dem/Val and EMD 

A difference was noted in the test program streamlining philosophies of managers 
from Natick and the other Commands. All Commands favored streamlining through 
concurrent or combined DT and OT/IOT&E. However, Natick managers favored 
combined testing, while managers from other Commands favored concurrent testing, in 
both Dem/Val and EMD. Combined testing entails greater program risk and Natick 
managers indicated a greater degree of risk acceptance. 

Additionally, it was concluded that the following techniques that address program 
documents and lower-level actions were used by most AC AT IV managers: 

1. Waive requirement for a COEA 

2. Waive requirement for a separate STAR 

3. Staff Milestone Decision Package prior to receipt of Independent 

Evaluation/Assessment Reports 

4. Use Interim Assessment or Evaluation to support a Milestone Decision 

It was concluded that ACAT IV programs generally do not require COEA or 
STAR program documents. The COEA is considered inaccurate, expensive and of little 
value. The STAR is generally unnecessary because most ACAT IV programs are not 
threat driven or combat related. They are typically combat support and combat service 
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support systems. Finally, independent evaluation and assessment repotting for Milestone 
Decision Reviews is commonly accelerated by staffing the Decision Review package 
prior to receipt of an evaluation/assessment or by utilizing an interim 
evaluation/assessment to support the Decision Review. Even though these techniques do 
not save much program time or funding, they represent low risk methods for acceleration 
of the process during the time-critical weeks prior to the Decision Reviews. 

Subsidiary Research Questions 2 and 3 - How do the various streamlining 
techniques compare with respect to program cost savings? and How do the various 
streamlining techniques compare with respect to program schedule reduction? The 
purposes of these segments of the study were to consider cost and schedule savings to: 
(1) determine if small and large size program differences existed, and (2) investigate 
contributing factors to the techniques cited above as having high usage. An additional 
intended purpose, quantification of the expected cost and schedule savings for the 
streamlining techniques, was not possible because of the low number of participants and 
respondents. 

The comparative analysis of differences between small and large size programs 
considered the cost savings differences for all techniques, as well as differences for the 
techniques with the highest and lowest cost and schedule reduction savings. Large 
programs were defined for purposes of the analysis as those requiring $7 million or more 
of RDTE funding (for the cost comparisons) and three years or more to complete 
development (for the schedule comparisons). The analysis concluded that cost savings for 
small ACATIV programs are equal to, or slightly lower than, large programs for almost 
every technique. There is close agreement concerning the relative cost savings associated 
with each of the streamlining techniques in the study. The two major program areas 
where large programs deviated from the trend may be due to the increased complexity 
of larger programs. As program size and complexity increases, actions such as IOT&E 
and contracting can grow dramatically, with associated greater potential for savings 
through streamlining. The two techniques were: 
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1. Combining EMD DT and IOT&E 

2. Using a single contract for Dem/Val and EMD 

In addition, it was concluded that large programs had a greater potential cost 
savings for urgency and threat related techniques. This indicates that larger programs 
are more likely to be for combat-related items, closer to higher AC AT levels, and more 
likely to require threat assessments and urgency-related acceleration due to a combat 
deficiency or enhancement. The specific techniques were: 

1. Request DA-Directed Procurement 

2. Waive requirement for a separate STAR 

The areas of analysis that compared the techniques with the highest usage and 
those with the highest and lowest savings supported the above conclusions for their high 
usage. The analysis contributing additional insights into the reasons for this high usage 
and the differences between small and large programs. The frequent use of the combining 
of milestones and the concurrency of testing phases by ACAT IV managers is supported 
by the high savings in program costs and time that is associated with these streamlining 
techniques. However, three techniques were noted as potentially saving a great deal, but 
they were not used often. They were: 

1. Combine Milestones I and III 

2. Combine EMD DT and IOT&E 

3. Conduct user evaluation in lieu of formal IOT&E in EMD 

The combining of Milestones I and III was concluded to be of limited application, 
to NDI or modified NDI programs only. The techniques of combining of DT and IOT&E 
and the use of a user evaluation in lieu of IOT&E were considered too risky and 
extremely difficult to implement, even for programs of the complexity of ACAT I Vs. 

The lowest saving techniques were evaluated because a number of them were both 
low in savings and high in use. The analysis of these techniques supports the reasons for 
their high use. It was concluded that techniques, such as the following, that save small 
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amounts of program time are important when their application supports key events, such 
as the Milestone Review: 

1. Staff Milestone Decision Package prior to receipt of Independent 

Evaluation/Assessment Reports 

2. Use Interim Assessment or Evaluation to support a Milestone Decision 

In addition, waivers of documentation, such as the COEA and STAR, do not 
create significant savings, but they are generally not applicable to, nor required of, 
AC AT IV programs. Therefore, it is concluded that waivers of these requirements are 
easy to obtain. 

Finally, it was concluded that some techniques, such as validation of the support 
system in production, are valuable because they provide a better product, not because of 
program funding or time savings. Some streamlining decisions are made based on quality 
considerations, a factor not considered in this study, where only cost and schedule were 
investigated. 

Commands’ ACAT IV Management Comparison - The final area of analysis 
in this study investigated the differences in the management of ACAT IV programs at the 
various AMC Commands. As noted earlier, the analysis of streamlining techniques 
portion of this study excluded programs managed by CECOM and MICOM, because all 
of their programs transitioned early in development to PEO management (and became 
ACAT III or higher). Although many programs at the other Commands transition to PEO 
management for development, there are some programs in all of the other Commands 
that remain within the management control of the AMC Commands, and some of these 
remain as ACAT IV (not requiring a certified PM). It was concluded that this difference 
in management control and responsibilities may be due to commodity differences. 
Commands such as Natick and ERDEC have whole commodity areas that have no related 
PEO, and all CECOM and MICOM commodity areas are directly linked to a PEO. The 
advent of the PEO system allowed for a complete transfer of control to the PEOs within 
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these Commands. This total control of the development programs in these commodity 
areas by the PEOs may be greater than was intended by the Packard Commission or DoD 
5000. From the management level selection criteria in the Army’s implementation 
regulation for DoD 5000 (AR 70-1) and the distinctions between the mission statements 
of AMC and the PEOs, it was concluded that the intent of DoD and the Army was that 
non-certified AMC managers manage the development of the smaller new systems in 
each of the commodity areas. However, PEO management does provide greater "Green- 
suit" input and this level of control could be more efficient if the PEOs did not create 
new organizations and certified PMs to manage these programs, and if they considered 
all avenues for meeting requirements, including the upgrade of fielded systems. 

Summary - ACAT IV programs have, and can be further streamlined from the 
standard DoD 5000 process across programs and Commands in a number of ways. The 
following recommendations include changes that should reflect better results than the 
standard process, for programs at this level of complexity. 

B. RECOMMENDATIONS 

1. Recommendations Regarding ACAT IV Management 

It is recommended that the results of this study be broadly distributed to ACAT 
IV managers. Even though the results provide no quantitative assistance for ACAT IV 
managers concerning expected savings levels for the various streamlining techniques 
investigated, the study does provide valuable information that should assist managers in 
their streamlining decision-making. One respondent commented, based on seeing only 
the survey form: 

You bring up lots of good ideas. I will keep a copy for use the next time 

I prepare an Acquisition Strategy. 

ACAT IV managers can benefit from having the list of possible streamlining 
techniques, from knowing which streamlining techniques are used most often, and from 
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knowing the factors, such as program risk, that they must consider when developing a 
streamlining strategy. 

ACATIV programs dramatically streamline the "standard" DoD 5000 acquisition 
process. The process is not the standard for them, but for ACAT I programs and, as 
such, the managers of the ACAT IV programs begin their program planning process by 
proposing significant changes from the "standard". This streamlining process requires the 
concurrence of many acquisition management team members, some of whom have no 
incentive to support the increased risk associated with streamlining. This can result in 
program delays while agreements and compromises are reached and significant 
unnecessary program costs and time following portions of the "standard" process that are 
unnecessary from a risk reduction perspective. Therefore, it is recommended that the 
U.S. Army revise the baseline development process for ACAT IV programs, to one that 
incorporates the key streamlining techniques of Combined Milestones I and II and 
concurrent testing in EMD that were concluded in this study as common to most ACAT 
IV programs. A comparison of the major phases and events and their differences for the 
two "standards" is at Figure 8. It is recommended that these techniques be established 
as standard procedures from which requirements are either increased or decreased. In 
addition, it is recommended that the alternative process exclude the requirements for 
COEA and STAR, except on an "exception basis" only. In place of the current system, 
where the manager must begin with the full ACAT I/II process and struggle with the 
acquisition team members to cut out elements, this recommendation would put risk-averse 
acquisition team members in a position of justifying the need for adding requirements 
versus the manager having to justifying their exclusion. 

Finally, it is recommended that the Army evaluate the ACAT IV management 
structure that is in place for CECOM and MICOM commodities. There is potential for 
over-management of the small programs in these commodity areas if they are all 
managed by certified PMs. There is a place for non-certified PM management (ACAT 
IV) in all commodities, as was intended by DoD 5000. The Army should consider 
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DoD 5000 Standard Acquisition Process 
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Figure 8. AC AT IV Acquisition Process Alternative 



establishing funding lines within AMC for CECOM and MICOM to manage the smallest 
development programs in their respective commodity areas. 

2. Recommendations for Further Research 

Because some of the ACAT IV managers contacted did not fully understand the 
DoD 5000 process and the standard terminology for acquisition programs, it is 
recommended that the training needs of ACAT IV managers be investigated. Note that 
ACAT IV level managers are not certified PMs and, as such, are not required to be as 
highly trained. However, because ACAT IV managers are typically civilians who do not 
have to rotate assignments every three years or less (as ACAT III Officers typically do), 
they may be fully effective as managers without additional training. Therefore, 
recommend a comparative analysis of the training requirements and experience levels of 
ACAT III (certified) and ACAT IV (non-certified) managers, to determine whether the 
experience and training of non-certified ACAT IV managers is adequate. 

Considering the lack of statistically significant results from this study, it is 
recommended that further studies be conducted that concentrate on the most significant 
techniques, that investigate more in-depth the establishment of useful statistics, such as 
confidence intervals, for the expected levels of cost and schedule savings. A major goal 
of this study that was unrealized was the establishment of planning estimates of expected 
cost and schedule savings, for ACAT IV manager streamlining decision-making. This 
type of information would be of great benefit to the U.S. Army ACAT IV development 
community. 

Based on the small number of ACAT IV managers in the U.S. Army and the 
Command differences regarding CECOM and MICOM (where all programs move up to 
at least ACAT III level), it is recommended that future research in this area consider both 
ACAT III and ACAT IV managers. The purpose would be twofold: (1) to study the 
differences between the two groups, and (2) if they prove to be similar, combine the two 
for statistical information concerning expected savings. 

Based on a recent briefing by the Commander of Marine Corps Systems 
Command, MG Mutter, the Marine Corps has many more ACAT III and IV programs 
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than AC AT I and II, and they are actively streamlining wherever possible, challenging 
all requirements of the DoD 5000 process. Their small size and shorter chain-of- 
command provide for an easier source of data than was available for this study of Army 
programs and may create a streamlining atmosphere that the Army could learn from. 
Therefore, it is recommended that further research be conducted that considers both 
Army and Marine Corps programs. 

Additionally, the following issues and streamlining techniques that were identified 
during the course of this study are recommended for more in-depth investigation: 

1. The reasons cited most often by ACAT IV managers for their streamlining 
decisions; which were risk level, and Command structure and commodity 
differences. 

2. The methods used for cross-fertilization of streamlining ideas among 
Commands. 

3. Since quality was a consideration in the selection of at least one streamlining 
technique, validation of the support system in first production instead of during 
EMD, further research into how quality impacts the acquisition streamlining 
decision-making process. 

4. Since procurement is as important to fielding of new systems as development, 
it is also recommended that ACAT IV procurement streamlining techniques be 
investigated in a separate, parallel effort.Included in this effort should be 
consideration of multi-year production contracting as a streamlining technique that 
can result in long-term cost and time savings. 

5. Investigation of capstone Program Management Documentation such as 
TEMPs, ORDs as a streamlining technique. This would allow for the 
development of several related efforts without waiting for separate staffing and 
creation of multiple documents. 
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APPENDIX A. ACAT IV POINTS OF CONTACT 


COMMAND 

NAME 

Natick RD&E Center 

Ken Rice 

Edgewood RD&E Center 

Dave LaBar 

Aviation Applied 
Technology Directorate 

Gene Birocco 

Missile Command 

Dr. Gene Paro 

Communications & 
Electronics Command 

Nancy Roberts 

Tank, Automotive and 
Armaments RD&E Center 

Jack Petersen 

Armaments RD&E Center 

Bill Blackwell 


PHONE NUMBER 
DSN 256-4883 
DSN 584-5272 
DSN 927-2822 

DSN 746-1457 
DSN 992-4159 

DSN 786-6411 

DSN 880-7044 
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APPENDIX B. ACAT IV MANAGER QUESTIONNAIRE 
Tailoring of the DoD 5000 Acquisition Process 

Data Required by the Privacy Act of 1974 


Prescribing Directive: AR 70-1 
Authority: 10 USC 4503 

Principle Purpose: The data collected by this questionnaire is to be used for 
research purposes only. 

Routine Uses: Full confidentiality of the responses will be maintained in the 
processing of this data. 

Mandatory or Voluntary Disclosure and Effect on Individual Not Providing 
Information: Your participation in this research is strictly voluntary. Individuals are 
encouraged to provide complete and accurate information in the interests of the 
research, but there will be no effect on individuals for not providing all or any part of 
the information. 


Questionnaire Instructions 

The purpose of this questionnaire is to gather data on how Army ACAT IV 
development managers tailor the DoD 5000 acquisition process. You have been 
selected to receive this questionnaire because you have served as an ACAT 
development manager since the implementation of the newest revision of the DoD 
5000 Series. 

- Please answer each question on the following pages. If you feel unqualified 
to answer a particular question, leave it blank. 

- Mark your answers directly on this questionnaire by placing an "X" over the 
appropriate box or writing in your response in the space provided. 

- When completed, please return the entire questionnaire in the pre-addressed 
government envelope provided. 

- Thank you. 
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A. Please estimate the following ORDERS OF MAGNITUDE totals for the project(s) that 
you have managed during the past 5 years (Total per project cost and duration estimates for 
the period from project initiation through the Milestone III decision). 

1. Development Cost: [o-ssooiq [$sook-$im] [$im-$3M] [$3m-s7M] [S7m-$ism] [sism-$3im] i>$3iv 

2 . Development Duration: {0-1 Years] [l-3 Years] [3-7 Years] P-15 Years] [> 15 Yen 


B. For each of the following potential acquisition process tailoring methods, answer the 
following three questions, followed by narrative comments concerning what positive or 
negative factors lead you to these selections: 


a. Have you or would you 
consider using this 
method? 

Yes No 

[Y] [N] 


b. How would you classify 
the cost savings potential 
of this method? 


SO- $100- S1K- S10K- >S100K 

$100 $1000 $10K 100K 

[1] [2] [3] [4] [5] 


c. How would you classify 
the schedule reduction 
potential of this method? 


0-7 1-4 1-6 6-18 >11 

Davs Wks Mos Mos Mos 

Hi 12] [3] (4] [5] 


1. Combine Milestones O and I (Skip a. [Y] [N] b. [1] [2] [3] [4] f5] c. [1] [2] [3] [4] ’5 

Concept Exploration Phase) 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

2. Combine Milestones I and II (Skip a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [41 f f 

Dem/Val Phase) 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 


3. Combine Milestones I and III (Skip a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] *5 

Dem/Val & EMD Phases) 

What are the key factors that lead you to the above decisions? 
a. 


b. 


c. 
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4. Combine Dem/Val development test (DT) a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 
and operational test (OT) 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

5. Conduct concurrent (overlapping) a. [Y] [N] b. [1] [2] [3] [4] [5] c. fl] [2] [3] [4) [5] 

Dem/Val DT and OT 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 


6. Combine EMD DT and Initial Operational a. [Y] [N] b. fl] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 
Test & Evaluation (IOT&E) 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 


7. Conduct concurrent EMD DT and IOT&E a. fY] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

8. Conduct Customer Tests in lieu of formal a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 
DT in Dem/Val Phase 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

9. Conduct Customer Tests in lieu of formal a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [31 [4] [5] 
DT in EMD Phase 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 
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10. Conduct user evaluation in lieu of formal a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4' 
IOT&E in EMD Phase 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

11. Request Milestone Decision Authority a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] K 

delegate Milestone Decision Review 
Chairmanship to RDEC Management 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

12. Request DA-Directed Procurement prior a. [Y] [N] b. [1] [ 2 ] [3] [4] [5] c. [1] [2] [3] [4' 
to Type Classification-Standard 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

13. Staff Milestone Decision package prior to a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] 
receipt of Independent Eval/Assess Repons 
(IER/IAR), with repons expected prior to the 
decision meeting. 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

14. Use Interim Assess or Eval Repons to a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] 

suppon a Milestone Decision, with decision 
contingent upon favorable Final Repons. 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 
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a. m [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 


15. Use an Independent Evaluator briefing in 
lieu of a report to support a Milestone 
Decision, with written report(s) to follow. 

What are the key factors that lead you to the above decisions? 
a. 


b. 


c. 

16. Waive requirement for a Cost and a. [Y] [N] b. [1] [2] [3] [4] [5] 

Operational Effectiveness Analysis (COEA) 

What are the key factors that lead you to the above decisions? 

a. 

b. 


c. 

17. Waive requirement for a separate System a. [Y] [N] b. [1] [2] [3] [4] [5] 
Threat Assessment Report (STAR) 

What are the key factors that lead you to the above decisions? 

a. 

b. 


c. 

18. Waive requirement for a separate a. [Y] [N] b. [1] [2] [3] [4] [5] 

Logistics Demonstration 

What are the key factors that lead you to the above decisions? 
a. 


b. 


c. 

19. Utilize TIWG as Electromagnetic a. [Y] [N] b. [1] [2] [3] [4] [5] 

Environ. Effects (E3) Requirements Board 

What are the key factors that lead you to the above decisions? 

a. 

b. 


c. 


C. [1] [2] [3] [4] [5] 


c. [1] [2] [3] [4] [5] 


c. [1] [2] [3] [4] [5] 


c. [1] [2] [3] [4] [5] 
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20. Conduct abbreviated Logistics Support a. [Y] [N] b. [1] [2] [3] [4] [5] c. 
Analysis and Reporting (LSA/LSAR) 

What are the key factors that lead you to the above decisions? 
a. 


b. 


c. 


21. Don’t prepare separate Configuration a. [Y] [N] b. [1] [2] [3] [4] [5] c. 

Management Plan 

What are the key factors that lead you to the above decisions? 
a. 


b. 


c. 

22. Don’t generate Independent Life Cycle a. [Y] [N] b. [1] [2] [3] [4] [5] c. 
Cost Estimate 

What are the key factors that lead you to the above decisions? 
a. 


b. 


c. 

23. Obtain Milestone Decision Authority a. [Y] [N] b. [1] [2] [3] [4] [5] c. 

approval to solicit a contract prior to 
Milestone Decision Review 

What are the key factors that lead you to the above decisions? 
a. 


b. 


c. 


24. Utilize a single contract for Dem/Val and a. [Y] [N] b. [1] [2] [3] [4] [5] c. 
EMD Phases 

What are the key factors that lead you to the above decisions? 
a. 


b. 


c. 


[1] [2] [3] [4] [5 


[1] [2] [3] [4] [f 


[1] [2] [3] [4] [f 


[1] [2] [3] [4] [5 


[1] [2] [3] [4] [5 
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25. Utilize a single contract for EMD and a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 

First Production 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

26. Conduct production First Article Tests to a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 
validate development deficiency corrections 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

27. Validate technical manuals/support a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 

system during First Production instead of 
during EMD 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

28. Other (specify) a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

29. Other (specify) a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 

30. Other (specify) a. [Y] [N] b. [1] [2] [3] [4] [5] c. [1] [2] [3] [4] [5] 

What are the key factors that lead you to the above decisions? 

a. 


b. 


c. 
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APPENDIX D. QUESTIONNAIRE NARRATIVE COMMENTS RAW DATA 


1. Combine Milestones 0 and I (Skip Concept Exploration (CE) Phase) 

Respondent 7: 

a. Items always of low technical risk and do not warrant concept exploration 
Respondent 9: 

a. Small programs usually don’t need to prove out technology to the extent of more complex 
ones 

Respondent 10: 

a. Most ACAT IV efforts have NDI or NDI-adaptation solutions 

b. The technology being considered is generally mature 

c. Milestone 0 has very little value added for ACAT IV efforts 
Respondent 11: 

a. Maturity of technology 

b. Airdrop testing costs 

c. Availability of test support (i.e. aircraft) 

Respondent 13: 

a. No, this is an essential phase of R&D 
Respondent 14: 

a. The potential success of a concept most of the time can only be proven out by trying to 
build a prototype 

b. The total manpower costs and schedule associated with doing a Milestone 0 will be 
somewhat costly. Today we are told that we need to deliver products in a few years, doing an 
additional step doesn’t streamline the process 

Respondent 16: 

a. By DoD 5000 Milestone 0 occurs when the MNS is approved. Because of time constraints 
almost all of our programs are started before the MNS is officially approved. Therefore 
technically we are in Dem/Val and our MNS is approved and available for Milestone I 
Respondent 17: 

a. Most ACAT IV items do not require a C.E. phase, most issues can be resolved in Dem/Val 
or EMD 
Respondent 21: 

a. Cost savings, schedule reduction 
Respondent 22: 

a. Not much difference between MS 0 and I. These days if your program is ready for MS 0, 
you’re probably ready for MS I 
Respondent 48: 

a. Basic technologies identified, moderate risk 

b. Need identified (MNS) 

c. Type of funding available (6.3, 6.4, and 6.5) 

Respondent 49: 

a. No, concept exploration is useful in determining system characteristics/configuration that 
meets the requirement, I feel it is needed 

b. Cost of this phase will not be reduced, but overall costs may be reduced 

c. Schedule in this phase will not be shortened, but overall schedule may be 
Respondent 60: 

a. NDI approach 

b. Cost of development bom by contractor 

c. Missing acquisition cycle step 
Respondent 63: 

a. Complexity of contemplated design change/improvement 

b. Evaluate associated risk in skipping the Concept Exploration Phase 
Respondent 64: 

a. Complexity of contemplated requirements, evaluate risk in skipping Concept Exploration 
phase 
Respondent 70: 

a. Contingent upon the clear, obvious demonstration of feasible concepts/technologies to be 
applied 

2. Combine Milestones I and II (Skip Demonstration/Validation (Dem/Val) Phase) 

Respondent 3: 

a. Shortens time and reduces number of documents, but allows for two generations of 
prototype development prior to MS III 
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Respondent 7: 

a. Items are typically of moderate technical risk and usually do not warrant Dem/Val prior to 
EMD 
Respondent 9: 

a. We are normally able to start building prototypes of the items we will test early on 

b. Funding for development is always in short supply, we usually have to prove out our chosen 
design while we are building the test items 

Respondent 10: 

a. Most ACAT IV efforts have NDI or NDI-adaptation solutions 

b. The technology being considered is generally mature, Milestone II is generally only useful 
for efforts with unproven technology 

Respondent 11: 

a. Maturity of technology, level of success during Concept Exploration 

b. Airdrop testing costs 

c. Availability of test support (i.e. aircraft) 

Respondent 13: 

a. This methodology seems to be taking place already 
Respondent 14: 

a. For our programs, seems like its a repetition of building a concept/prototype and then test 

b. Cost ana time constraints 
Respondent 16: 

a. NDI solution was not acceptable because after early DT testing the ORD requirements were 
not met. Therefore, the program was changed from I/III milestone decision (one milestone) to 
the next which is I/II and III (two milestones) 

Respondent 17: 

a. Most ACAT IV items do not require a C.E. phase, most issues can be resolved in Dem/Val 
or EMD 
Respondent 21: 

a. Cost savings, schedule reduction 
Respondent 22: 

a. We did it 
Respondent 48: 

a. Basic technologies identified, low risk 

b. Need identified (MNS) 

c. Type of funding available (6.3 and 6.5) 

Respondent 49: 

a. Off-the-shelf items or even items that have been developed and are low risk do not need to 
follow an extensive Dem/Val, I would consider skipping 

b. The cost will be reduced because you use off-the-shelf items 

c. The schedule will be shortened because off-the-shelf items can be procured quickly 
Respondent 60: 

a. NDI approach 

b. Cost of development bom by contractor 

c. Missing acquisition cycle step 
Respondent 63: 

a. No, Dem/Val is essential to the development effort 

b. Unnecessaiy to assume risk when test costs are relatively small 

c. Test costs following the Dem/Val Phase escalate greatly 
Respondent 64: 

a. No, no reason to assume risk when test costs are low, after Dem/Val phase costs of tests 
are high 
Respondent 67: 

a. We are in fact using this approach. Procurement lead time savings are substantial 
Respondent 70: 

a. Contingent upon the verifiable, creditable demonstration of technologies considered, must 
take into account the level of maturity of given technology/solution 

3. Combine Milestones I and El (Skip Dem/Val and Engineering & Manufacturing Development 
(EMD) Phases) 

Respondent 7: 

a. Items are sometimes simple and pose little risk by going directly to Milestone III 
Respondent 9: 

a. This is my favorite and we are doing it for the Modular General Purpose Tent System. 
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Since we normally only have one shot at building items we can normally do a MS I/III 

b. It saves test and prototype costs 

c. It lets us do 2-3 year programs vs. 5-7 years, very important to cut time out, the user is 
demanding quicker turn-around 

Respondent 10: 

a. Done it 

b. The maturity of most AC AT IV technology lends itself to pursuing a combined Milestone 
I/III 

c. P3I efforts can fix the few issues left unresolved after Milestone III 
Respondent 11: 

a. Maturity of technology, documented early successful testing 

b. Airdrop testing costs are high 

c. Availability of test assets 
Respondent 13: 

a. No, by skipping the EMD Phase, your R&D effort would be left up to the contractor once 
you go to production. The Army would not know the history of their own programs 
Respondent 14: 

a. Not applicable to all systems, combining the two for an NDI makes sense. For a non-NDI 
need to have a check point in between Milestones I and II to make sure all IPR members sign 
on to a concept. In light of going away from drawings/specs, on the other hand, why have 
extra milestones if you are Type Classifying a concept. Obviously, eliminating some schedule 
will save time but could result in money being put into a dead end if nobody likes the concept. 
Respondent 16: 

a. This is done for all NDI solutions. This is the first thing we look at for our programs, 
because the 5000 series requires NDI solution before any government development. This also 
allows the fastest and cheapest development which is important for ACAT IV programs 
Respondent 17: 

a. No, I think you always need to do EMD to prove-out technical issues 
Respondent 22: 

a. No, too much rushing of development 
Respondent 48: 

a. Basic technologies identified, probably NDI/modified NDI 

b. Need identified (MNS and ORD) 

c. Type of funding available 
Respondent 49: 

a. Off-the-shelf or low risk technology are ideally suited for combining these phases because 
little Dem/Val or EMD effort is needed. I am currently combining I and III for an ACAT IV 

E rogram 

. Cost will be reduced because of reduction in work 
c. Schedule will be shortened also 
Respondent 60: 

a. No, disaster lurks - key stop is producibility and safety for armaments, must do EMD 
Respondent 63: 

a. No, Dem/Val is essential to the development effort 

b. Unnecessary to assume risk when test costs are relatively small 

c. Test costs following the Dem/Val Phase escalate greatly 
Respondent 64: 

a. No, no reason to assume risk when test costs are low, after Dem/Val phase costs of tests 
are high 
Respondent 67: 

a. No, unacceptable program risk with this approach. Large number of test quantities in EMD 
discourage elimination of EMD Phase 
Respondent 69: 

a. Up to 5 contractors had non-developmental items potentially capable of meeting 
requirements in performance spec and ROC. Contractors were willing to provide 3 bid samples 
for testing, at no cost to the government, as part of the solicitation process, leading to a 
multiyear (5 year) production contract. Milestone III production decision IPR judged the 
acceptability of the contractors test results prior to award. Contract obtaining both production 
hardware and data for transition to organic support after 2 - 3 years of interim contractor 
support. Potential for cost savings was great aue to competition 
Respondent 70: 

a. In support of an NDI acquisition, our team was able to combine Milestones I and III rather 
effectively with total management support, this streamlining or tailoring decision was a 
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common sense approach as the hardware was developed and available as a commercial off-the- 
shelf 


4. Combine Dem/Val Development Test (DT) and Operational Test (OT) 

Respondent 3: 

a. DT with user troops is sufficient to identify and verify user issues 
Respondent 7: , „ 

a. DT/OT is a reasonable alternative for relatively simple items with few operational issues, 
small emphasis on OT 
Respondent 9: 

a. Time saver, cost saver on number of prototypes needed, testing, and reporting 

b. Often difficult to get OEC and TEXCOM to agree 
Respondent 10: 

a. Testing on ACAT IV efforts is often overdone 

b. Proper engineering work can limit required testing 

c. Many DT issues can be looked at during a thorough OT 
Respondent 11: 

a. Time, maximize return from testing costs 

b. Airdrop testing costs 

c. Length of an airdrop OT 
Respondent 14: 

a. If TECOM does it, yes. TEXCOM, if in the lead, is extremely costly and manpower 
intensive and doesn’t do well in measuring technical parameters. Our RAM driven tests are 
extremely similar to TEXCOM’s OT: Mission profiles are followed. OT tests for CBPS cost 
approximately $700K versus one DT test is approximately $150K 
Respondent 16: 

a. Cost 

b. Time, because the ORD has to be approved before you can be in the TSARC (which you 
have to be in a year before OT) which is never the case here 

Respondent 17: 

a. Makes a lot of sense, there is often overlap between technical and operational issues, 
separate tests stretch-out the schedule too much 
Respondent 21: 

a. No, it will be too costly in case of gross failures 
Respondent 22: 

a. No, too hard to schedule OT troops 
Respondent 48: 

a. Degree of testing required 

b. Ability to simultaneously collect technical and operational data 

c. Ability to get concurrence from DT and OT testers/evaluators 

Respondent 49: , „ , , , t 

a. Off-the-shelf items should not need very much DT/OT testing because of the low technology 
risks. I have considered this but it is difficult to convince the testing folks that testing is not 
required or should be limited 

b. Cost of testing is very expensive 

c. Schedule can be greatly shortened, in some situations the acquisition cycle can be cut in half 
Respondent 60: 

a. No, can’t man-fire during Dem/Val to support OT 
Respondent 63: _ , 

a. Combining DT/OT will make testing more cost effective, i.e., save time and funds 

Respondent 64: 

a. Save cost, save time 
Respondent 67: 

a. Artillery ammunition typically does not require OT 


Respondent 69: „ J , 

a. Bid sample testing at APG demonstrated system performance and also conducted 
environmental testing which was the equivalent of DT testing. Combat Developer agreed to 
conduct an operational evaluation of the system during bid sample testing so they could 
provide an independent assessment for the IPR 

Respondent 70: . r . . . 

a. No, we as developers may be biased when considering the performance/efiectiyeness or 
systems under development. Truly recommend maintaining an independent operational test or 
user test, customer test, etc.. This provides developers a fresh perspective from customers 
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eyes (not just user representatives) 

5. Conduct concurrent (overlapping) Dem/Val DT and OT 
Respondent 7: 

a. Only useful as a means of accelerating test schedule 
Respondent 9: 

a. OK, but probably not worth fighting for, this wouldn’t save you that much 
Respondent 10: 

a. Program dependent - easy to accomplish with smaller items and many prototypes 
Respondent 11: 

a. Doesn’t maximize return on testing costs, only saves time. Would conduct only if directed 
to 

b. There would be no savings because you are duplicating efforts 

c. The concurrent nature would save time 
Respondent 14: 

a. No, if any problems discovered at DT and you can’t correct them, you are in trouble in 
OT. If the system is doing poorly in RAM the test will be terminated. OEC wants production 
representative systems 
Respondent 16: 

a. Try to remove operational testers from the process, because of cost and time. Also they 
make the AC AT IV program act like a AC AT I, which we can never do because of time and 
money 
Respondent 17: 

a. Separate tests take too long 

b. Overlap of DT and OT issues 

c. May require more test items than are affordable 
Respondent 22: 

a. Any concurrency is good 
Respondent 48: 

a. Degree of testing required 

b. Ability to simultaneously collect technical and operational data 

c. Ability to get concurrence from DT and OT testers/evaluators 
Respondent 49: 

a. One test program should meet the testing requirements for the program, this idea should be 
the way all testing should be done, currently it is almost impossible to conduct one test 

b. Costs would be reduced, one test would cost far less than three separate tests 

c. Schedule would be greatly shortened 
Respondent 60: 

a. No, can’t man-fire during Dem/Val to support OT 
Respondent 63: 

a. Combining DT/OT will make testing more cost effective, i.e., save time and funds 
Respondent 64: 

a. Save cost, save time 
Respondent 70: 

a. Definitely support this, DT/OT not necessarily dependent on each other. Concurrent testing 
will bring you to completion quicker, shorten project schedule, save labor dollars 

6. Combine EMD DT and Initial Operational Test & Evaluation (IOT&E) 

Respondent 3: 

a. DT with user troops is sufficient to identify and verify user issues 
Respondent 9: 

a. Time saver, cost saver on number of prototypes needed, testing, and reporting 

b. Often difficult to get OEC and TEXCOM to agree. Keep in mind that Natick does not 
always do separate Dem.Val, OT, EMD, and IOT&E even now 

Respondent 11: 

a. Prior success rate of testing which would reduce risk and increase confidence for success in 
the combined effort 

b. Airdrop testing costs 

c. Normal IOT&E time allotments for airdrop 
Respondent 14: 

a. ? 

Respondent 16: 

a. Start with NDI solution so most of our testing is this type DT combined with IOT&E 
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Respondent 17: 

a. Makes a lot of sense, there is often overlap between technical and operational issues, 
separate tests stretch-out the schedule too much 
Respondent 21: 

a. Cost savings, schedule reduction 
Respondent 22: 

a. No, too hard to schedule troops 
Respondent 48: 

a. Degree of testing required 

b. Ability to simultaneously collect technical and operational data 

c. Ability to get concurrence from DT and OT testers/evaluators 
Respondent 49: 

a. One test program should meet the testing requirements for the program, this idea should be 
the way all testing should be done, currently it is almost impossible to conduct one test 

b. Costs would be reduced, one test would cost far less than three separate tests 

c. Schedule would be greatly shortened 
Respondent 60: 

a. IOT&E conducted as part of EMD, following man rating 
Respondent 64: 
a. ? 

Respondent 70: 

a. No, we as developers may be biased when considering the performance/effectiveness of 
systems under development. Truly recommend maintaining an independent operational test or 
user test, customer test, etc.. This provides developers a fresh perspective from customers 
eyes (not just user representatives) 

7. Conduct concurrent (overlapping) EMD DT and IOT&E 
Respondent 9: 

a. OK, but probably not worth fighting for. This wouldn’t save you that much 
Respondent 11: 

a. Only if forced to, results in time savings only 

b. Results in no cost savings, because testing still must be completed 

c. The concurrent nature will save time 
Respondent 14: 

a. ? 

Respondent 16: 

a. No, if you try to involve the operational testers you will have big problems with money, 
time, and test constraints 
Respondent 17: 

a. Separate tests take too long 

b. Overlap of DT and OT issues 

c. May require more test items than are affordable 
Respondent 21: 

a. Cost savings, schedule reduction 
Respondent 22: 

a. Concurrency is good (if you have the manpower to observe both) 

Respondent 48: 

a. Degree of testing required 

b. Ability to simultaneously collect technical and operational data 

c. Ability to get concurrence from DT and OT testers/evaluators 
Respondent 49: 

a. One test program should meet the testing requirements for the program, this idea should be 
the way all testing should be done, currently it is almost impossible to conduct one test 

b. Costs would be reduced, one test would cost far less than three separate tests 

c. Schedule would be greatly shortened 
Respondent 60: 

a. IOT&E conducted as part of EMD, following man rating 
Respondent 64: 
a. ? 

Respondent 70: 

a. Definitely support this, DT/OT not necessarily dependent on each other. Concurrent testing 
will bring you to completion quicker, shorten project schedule, save labor dollars 
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8. Conduct Customer Tests in lieu of a formal Development Test in Dem/Val Phase 

Respondent 9: 

a. Good idea, gets the customers input up front, TIWG may be a tough sell on this 

b. Not a huge cost saver up front, but could identify problem areas early which could save 
money 

Respondent 10: 

a. Definite advantages for very mature technologies and NDI efforts 
Respondent 11: 

a. If technical evaluators allow, it provides flexibility in selecting test sites, i.e. other than 
TECOM assets 

b. Looking for lower cost test support 

c. New test site could also save time if private sector 
Respondent 13: 

a. No, R&D DT is very different from Customer Testing 
Respondent 14: 

a. No difference, cost and schedule the same. The only thing to worry about is that will 
TECOM accept the results of a customer test 
Respondent 16: 

a. Speed, this allows for a quick test, short lead time 

b. Can send prototypes that are not 100% ready, also can send technicians to support test, 
have more direct control of test 

Respondent 17: 

a. Good idea, but the current philosophy of the testers and evaluators is that customer tests 
don’t count, i.e., can’t use results for type classification decision 
Respondent 21: 

a. No, you need to know if system works before giving it to the troops 
Respondent 48: 

a. Degree of testing required 

b. Capabilities of Government test facilities vs. industry and cost differential 
Respondent 49: 

a. This method is more customer oriented so you know exactly if your product meets the users 
requirements, the test agency may not have an appreciation for the critical issues and issues 
that are not that critical 

b. Costs would be reduced 

c. Schedule would be greatly shortened 
Respondent 60: 

a. No, little advantage, must keep comm, involved, leads to formal DT/TT with TEMP 
Respondent 63: 

a. No, conduct of formal Dem/Val DT reduces risk of future formal test efforts 
Respondent 64: 

a. No, formal DT in Dem/Val, reduce risk 
Respondent 67: 

a. No, artillery requires safety tests 
Respondent 70: 

a. No, I considered operational testing same as customer tests, if OT is not structured as a 
customer test then it should be. In any event conducting customer tests in place of DT may not 
allow us to measure system effectiveness without imposing lengthy customer tests (i.e., years) 

9. Conduct Customer Tests in lieu of a formal Development Test in EMD Phase 
Respondent 9: 

a. I think we have to get formal, structured testing sometime 

b. I think customer testing this late in the design process might create chaos 
Respondent 10: 

a. Definite advantages for very mature technologies and NDI efforts 
Respondent 11: 

a. If technical evaluators allow, it provides flexibility in selecting test sites, i.e. other than 
TECOM assets 

b. Looking for lower cost test support 

c. New test site could also save time if private sector 
Respondent 14: 

a. No difference, cost and schedule the same. The only thing to worry about is that will 
TECOM accept the results of a customer test 
Respondent 16: 
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a. No, at this stage of program it is hard to substitute customer tests for more formal DT by 
TECOM. Also, would get better information from TECOM test 
Respondent 17: 

a. Good idea, but the current philosophy of the testers and evaluators is that customer tests 
don’t count, i.e., can’t use results for type classification decision 
Respondent 22: 

a. No, still need independent tester/evaluator 
Respondent 48: 

a. Degree of testing required 

b. Capabilities of Government test facilities vs. industry and cost differential 
Respondent 49: 

a. This method is more customer oriented so you know exactly if your product meets the users 
requirements, the test agency may not have an appreciation for the critical issues and issues 
that are not that critical 

b. Costs would be reduced 

c. Schedule would be greatly shortened 
Respondent 60: 

a. Little advantage, must keep comm, involved, leads to formal DT/TT with TEMP 
Respondent 63: 

a. No, conduct of formal Dem/Val DT reduces risk of future formal test efforts 
Respondent 64: 

a. No, formal DT in Dem/Val, reduce risk 
Respondent 67: 

a. No, artillery requires safety tests 

Respondent 70: „ . 

a. No, I considered operational testing same as customer tests, if OT is not structured as a 
customer test then it should be. In any event conducting customer tests in place of DT may not 
allow us to measure system effectiveness without imposing lengthy customer tests (i.e., years) 

10. Conduct user evaluation in lieu of formal IOT&E in EMD Phase 

Respondent 9: 

a. This is the whole point of IOT&E 

b. Might get better information from real users 
Respondent 11: 

a. If the operational evaluators approve of an alternative test site 

b. Seek out lowest testing costs 

c. May save time 
Respondent 13: 

a. Customer focus, see what the user wants/needs not the operational people at OEC 
Respondent 14: 

a. Formal IOT&E are extremely expensive, need all logistics products (i.e. LSA reports, 
RPSTLs, maintenance tasks, training, TMs) fully developed 
Respondent 16: . , 

a. Try to conduct any testing that substitutes for formal OT because of the operational testers 
treatment of ACAT IV programs 
Respondent 17: 

a. Makes sense for most Natick items. Formal OT should be for large, complex systems 
Respondent 21: 

a. Cost savings 
Respondent 22: 

a. Have yet to participate in well organized OT 
Respondent 48: 

a. Degree of testing required 

b. Capabilities of Government test facilities vs. industry and cost differential 
Respondent 49: 

a. This method is more customer oriented so you know exactly if your product meets the users 
requirements. The test agency may not have an appreciation for the critical issues and issues 
that are not that critical 

b. Costs would be reduced 

c. Schedule would be greatly shortened 
Respondent 60: 

a. No, need formal IOT&E to get OEC support for milestone 
Respondent 64: 
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a. No, formal DT in Dem/Val, reduce risk 
Respondent 70: 

a. The intent of any OT should be user evaluation. See no cost/time benefit. However will 
allow for a system truly reflecting user requirements. We need to focus and listen to user to 
meet their expectations 

11. Request Milestone Decision Authority delegate Milestone Decision Review Chairmanship to 
RDEC Management 

Respondent 3: 

a. Much quicker and simpler 
Respondent 9: 

a. Empowerment 

b. Potential for quick turn-around is unlimited 

c. Type Classify when the system is ready, not just when the bureaucracy is satisfied and has 
spent all the money 

Respondent 10: 

a. The MDR chairmanship should be at the lowest possible working level 
Respondent 11: 

a. Time savings, ease of coordination with an on-site activity 

b. Eliminate travel costs 

c. Reduces coordination time 
Respondent 13: 

a. RDEC management will know your program - where an MDA is just signing on the dotted 
line 

Respondent 14: 

a. No advantage in cost and schedule, having it in the RDEC aids in that the MDA can be 
more involved in the program when in the RDEC. Now the MDA only gets involved in high 
visibility programs 
Respondent 16: 

a. No, current change in Natick has the MDA being established at Natick because of the 
starting of SSCOM 
Respondent 17: 

a. Current MDA at ATCOM is not close enough to the development program 
Respondent 22: 

a. No, like having general officer sign it 
Respondent 48: 

a. No, see no benefit 
Respondent 49: 

a. No, the materiel developer knows the requirement for fielding items and he is the manager 
of that item when it is in the field, RDEC should not be responsible for making that decision 
Respondent 60: 

a. System being non-complex 

b. Schedule streamlining 

c. Precedence set previously 
Respondent 64: 

a. ? 

Respondent 67: 

a. With new command, ARDEC has not yet got this authority 
Respondent 69: 

a. No, production decision IPR package (signed by CDR, ARDEC) was assembled with test 
reports and development and operational independent assessments supporting a Type 
Classification Standard recommendation and recommending award of the multi-year production 
contract 
Respondent 70: 

a. This is being done today, thus have not implied time/cost benefit 

12. Request DA-Directed Procurement prior to Type Classification-Standard 

Respondent 9: 

a. No, confuses issues 

b. More bureaucracy 

c. I may be missing some finer points on this one 
Respondent 10: 

a. No, considered it - actually a very difficult task, over 18 different wickets to go through, 
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not an effective method under current system 
Respondent 11: 

a. Only if the customer/user has an immediate need 

b. Costs will still be incurred to achieve the Type Classification 

c. Time savings to initial fielding will be achieved, not Type Classification 
Respondent 14: 

a. Probably no change in cost and schedule. With no more specs, we could maybe combine 
RDTE into the production phase. The advantage of having DA directed procurement is that 
with money in production it helps keep funds in RDTE program, ensures user and IPR 
members support. Part of our streamlining theory that we shall not develop anything the Army 
doesn’t want to buy 
Respondent 16: 

a. No, have never done this for any of my programs and can not see any advantages 
Respondent 17: 

a. This was done on SICPS tent, needed to get items fielded even though RDTE was not 100% 
complete. Makes sense because the soldier would rather have an 85% solution now than wait 2 
years for the 100% solution 
Respondent 21: 

a. Prove out technical data package 
Respondent 22: 

a. No, can’t see why this would be a good idea 
Respondent 48: 

a. No, see no benefit 
Respondent 49: 

a. No, no benefit 
Respondent 60: 

a. No, Program directed competition 
Respondent 64: 
a. ? 

Respondent 70: , 

a. No, This is inconsistent with shelving technology (i.e., develop system, finalize TDP, and 
have available for procurement with no POM forecasts) 

13. Staff Milestone Decision Package prior to receipt of Independent Evaluation/Assessment 
Reports (IER/IAR), with reports expected prior to the decision meeting 
Respondent 3: 

a. This will speed up program substantially 
Respondent 9: 

a. Time saver! 

b. Tough sell to TIWG/JWG 
Respondent 10: 

a. Gets much of the staff review accomplished without waiting for late test reports, has worked 
well in the past for us 
Respondent 11: 

a. Level of confidence the favorable reports will receive 

b. Travel costs 

c. Time saved from holding an additional one or two meetings 
Respondent 13: 

a. This would familiarize the MDA with your program 
Respondent 14: 

a. If you have to meet a Type Classification date, the evaluators can just present their 
assessments at the IPR provided they tell you their issues beforehand so there are no surprises. 
If you have the reports the advantage is that maybe you can avoid the IPR and just do it by 
correspondence 
Respondent 16: 

a. Time, need the MDS III to be done in time to start the production procurement actions 
(transition to ATCOM). Delays could cost loss of procurement funding 
Respondent 17: 

a. Final reports take too long, results are usually known long before final report 
Respondent 21: 

a. Time savings 
Respondent 22: 

a. Pretty risky - are IPR participants willing to bring positions to meeting without final 
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reports? 

Respondent 48: 

a. Shorten milestone decision process 
Respondent 49: 

a. I would try this method to expedite the Milestone Decision Reviews, although the IER/IAR 
are usually received in a timely fashion 

b. Costs are not significant 

c. Sometimes it could shorten the schedule significantly 
Respondent 60: 

a. Smart business decision, low risk 
Respondent 63: 

a. Staffing prior to receipt of IER/IAR will reduce response time for Type Classification 
position statement 
Respondent 64: 

a. Reduce response time 
Respondent 69: 

a. No, preliminary staffmg was done with preliminary assessments but final assessments were 
available for formal Milestone III IPR production decision 
Respondent 70: 

a. Manager should have clear knowledge of DT/OT test results and willing to accept 
responsibility for IARs content, this action saves time 

14. Use Interim Assessment or Evaluation Reports to support a Milestone Decision, with decision 
contingent upon favorable Final reports 
Respondent 3: 

a. This will speed up program substantially 
Respondent 9: 

a. More time saved 

b. Gives assessor/evaluator time to do the reports right 
Respondent 10: 

a. Definite time saver with little risk 
Respondent 11: 

a. Provided the data in the interim reports will be favorable for the decision 

b. Little to no cost savings 

c. Time saved from waiting for the final reports 
Respondent 13: 

a. This would keep the decision process moving forward instead of standing still waiting for 
final reports 
Respondent 14: 

a. Sounds risky, if the IPR signs up to something, they should stick to it. Since the final 
decision isn’t made until final reports are there, what is the point 
Respondent 16: 

a. Time, need the MDS III to be done in time to start the production procurement actions 
(transition to ATCOM). Delays could cost loss of procurement funding 
Respondent 17: 

a. Usually, not much changes between the interim and the final report 
Respondent 21: 

a. Cost/time savings 
Respondent 22: 

a. Great idea if evaluators are willing to do so. Currently an interim report is just as hard to 
get as final so we push for final 
Respondent 48: 

a. Shorten milestone decision process 
Respondent 49: 

a. I would try this method to expedite the Milestone Decision Reviews, although the IER/IAR 
are usually received in a timely fashion 

b. Costs are not significant 

c. Sometimes it could shorten the schedule significantly 
Respondent 60: 

a. Smart business decision, low risk 
Respondent 63: 

a. Use of interim reports/with contingency statement will reduce response times 
Respondent 64: 
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a. Do most of the decision pending receipt of final report 
Respondent 70: ... _ , 

a. Sounds reasonable as long as evaluator is willing to bless off on an interim assessment that 
conveys his/her position 


15. Use Independent Evaluator Briefing in lieu of a reports to support a Milestone Decision, with 
written report(s) to follow 

Respondent 3: 

a. This would be the quickest method 
Respondent 9: 

a. Common sense, MDR should believe evaluator 

b. More and more time saved 

c. Gets item to the field quicker 
Respondent 10: 

a. Hadn’t considered - but a good idea 
Respondent 11: 

a. Convenience/time 

b. Shouldn’t affect cost 

c. Save the time that would be lost waiting for the final report 

Respondent 14: _ . 

a. If you have to meet a Type Classification date, the evaluators can just present their 
assessments at the IPR provided they tell you their issues beforehand so there are no surprises. 
If you have the reports the advantage is that maybe you can avoid the IPR and just do it by 
correspondence. They can’t be allowed to change their position in the report versus briefing 

Respondent 16: . . , . ^ r . 

a. They normally attend milestone decision and give this briefing no matter the status oi the 

evaluation 


Respondent 17: , _ 

a. The important thing is to get the information to the IPR and MDA, if a briefing can 
accomplish this why wait for the written report? 

Respondent 22: ^ . _ 

a. Sounds simpler, but how do you ensure loop is closed i.e., receipt of reports? 

Respondent 48: 

a. Shorten milestone decision process 


Respondent 49: 

a. No, I don’t see benefit 
Respondent 60: 

a. Independent evaluator and key team member 
Respondent 63: 

a. Only if the milestone decision requires a formal briefing that 
correspondence 
Respondent 64: 

a. If well documented 


could not be accomplished by 


Respondent 70: . _ ______ 

a. No, I consider an interim assessment for milestone decision but not briefing. Why ? - May 
circumvent accountability - what if independent evaluator is not available (sick, training), 
program momentum could require a delegated representative or stand-in convey critical 
position with decreasing knowledge and accountability 


16. Waive requirement for a Cost and Operational Effectiveness Analysis (COEA) 
Respondent 3: 

a. Unnecessary for ACAT IV programs 
Respondent 9: _ 

a. I don’t know, when would cost analysis be done and by whom? 

b. Incumbent on the developer to have figures developed 

c. Empowerment 
Respondent 10: 

a. Very expensive and generally useless document 


Respondent Qn the p rocuremen t numbers associated with the effort, if the system requirement 
is not strong, the document may prove beneficial when selling the program 


b. Cost to create the document 


c. Time saved from preparing the document 
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Respondent 14: 

a. Don’t think we do this for ACAT IVs 
Respondent 16: 

a. The cost for a COE A is usually equal to the total cost for the whole RD&E effort. 
TRADOC is deciding if all ACAT Ills and IVs will be excluded from the COEA requirement 
Respondent 17: 

a. It is impossible to be very accurate. This applies mostly to large, high cost systems 
Respondent 22: 

a. Haven’t had a COEA done that reflects realistic development item usage, COEA personnel 
unable to become familiar with item 
Respondent 48: 

a. Relative cost of product, degree of user support 
Respondent 49: 

a. No, this document is needed to determine cost savings which is a major part of the decision 
process. It also provides information for budgeting purposes 
Respondent 60: 

a. No, need to assure that investment is smart 
Respondent 63: 

a. No, this should be a TRADOC decision 
Respondent 64: 

a. No, this is the training school decision 
Respondent 70: 

a. Conceptually, COEAs are great. They allow decision maker to justify their actions and 
developmental efforts. Unfortunately the level of effort involved for a small program can 
reduce any if not all benefits. Do not see as benefit to customer for small dollar programs - 
only value to manager for justifying 


17. Waive requirement for a separate System Threat Assessment Report (STAR) 
Respondent 9: 
a. ? 

Respondent 10: 

a. Most items at Natick have no impact on threat 
Respondent 11: 

a. If the system will be in a limited or no combat environment 

b. Cost to create the document 

c. Time saved from preparing the document 
Respondent 14: 

a. Don’t think we do this for ACAT IVs 


Respondent 16: 

a. Not reauired for ACAT IV 


Respondent 22: 

a. Especially for ACAT IV, another big group of people that can’t take time to know item 
Respondent 48: 

a. Generally not needed or applicable for aviation ground support equipment 
Respondent 49: 

a. I do not understand the requirement for this report 
Respondent 60: 

a. Common sense 
Respondent 64: 
a. ? 


Respondent 69: 

a. Determination made that this was not required 


18. Waive requirement for a separate Logistics Demonstration 
Respondent 3: 

a. Logistics issues can be determined in DT 

Respondent 9: . 

a. We are doing this now, it works. The log folks have plenty of opportunities to input their 
information requirements at TIWGs 
Respondent 10: 

a. A very expensive and generally unnecessary event for most ACAT IV efforts 
Respondent 11: 

a. Yes, if during operational testing data was taken to show usability and maintainability of the 
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product 

b. Cost to set up test and TDY 

c. Time saved from setting up and holding the log demo 
Respondent 14: 

a. Only if the TIWG concurs, particularly OEC and schools that will do maintenance on your 
system, depends on complexity of system 
Respondent 16: 

a. Usually include any Log Demo requirements in the First Article Tests because that is 
usually the first time the system is produced according to the spec 
Respondent 17: 

a. Should be done as part of the OT or DT/OT 
Respondent 48: 

a. Aviation ground support equipment is logistics-related equipment, so Log Demo may be 
redundant testing 
Respondent 49: 

a. Logistics for off-the-shelf items are limited, also the contractor can provide logistics support 

b. Cost of a Logistics Demonstration can be eliminated 

c. Schedule can be reduced 
Respondent 60: 

a. Not required for ammo items 
Respondent 64: 
a. ? 

Respondent 69: 

a. Logistics Demonstration was not required during bid sample testing. The successful 
contractor is providing complete logistics inputs under contract CDRL requirements 

19. Utilize Test Integration Working Group (TIWG) as Electromagnetic Environmental Effects 
(E3) Requirements Board 

Respondent 9: 
a. ? 

Respondent 10: 

a. Not relevant to my programs 
Respondent 11: 

a. If I can do it saving time and money I will, but I currently have little understanding of the 
E3 Requirements Board 
Respondent 14: 

a. The E3 Board doesn’t make any sense, particularly for our programs 
Respondent 16: 

a. I think this is now a requirement for us 
Respondent 22: 

a. Need E3 requirements spelled out clearly in DoD 5000 or other regulations, TIWG must 
become familiar with E3 if this is to work, ARDEC is unmatched in their knowledge 
Respondent 48: 

a. We use TIWGs to manage all aspects of test program 
Respondent 49: 

a. Never addressed in our programs 
Respondent 60: 

a. No, members of E3 Board are subset of TIWG 
Respondent 64: 
a. ? 

Respondent 69: 

a. TIWG and TEMP approved the EMI testing conducted. EMP testing may be conducted for 
information only on First Article Test sample 
Respondent 70: 

a. TIWGs define test measures that allow us to assess performance of a system versus user 
requirement. In this we ensure systems are safe, effective, environmentally compatible, do not 
impose health hazards, survivable, etc. No reason to have E3 separate from a group of users, 
independent evaluators, logisticians, etc. whom bring already independent views and analysis 

20. Conduct abbreviated Logistics Support Analysis and Reporting (LSA/LSAR) 

Respondent 9: 

a. ? I don’t know enough about this 
Respondent 11: 
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a. Yes, if the system is not too complex for the user to operate and maintain 
Respondent 14: 

a. Can conduct abbreviated reports only if TIWG, TRR bodies agree, particularly OEC, user 
and maintenance schools. If don’t have their approval, won’t get training certification for OT 
and the test won’t happen 
Respondent 16: 

a. If a LSA is required it is tailored to the program 
Respondent 48: 

a. We do aviation ground support equipment, not major systems. LSA is tailored accordingly 
Respondent 49: 

a. Logistics support is very limited for off-the-shelf items, contractors provide support also 

b. Cost can be reduced appropriately 

c. Schedule can be shortened appropriately 
Respondent 60: 

a. No, LSAVLSAR not required for ammo 
Respondent 64: 

a. ILS manager to make decision 
Respondent 69: 

a. No, no LSAR inputs were obtained as part of bid sample testing, but successful contractor 
is doing full LSA as part of contract CDRL requirements 
Respondent 70: 

a. Yes, this area needs emphasis. As the LSA/LSAR analysis and reporting was based on 
major systems, I have found that mis-proportionate amounts of emphasis is placed on small 
items required little logistic support. Need common sense approaches - why have five logistics 
persons supporting a four member project office team? 

21. Don’t prepare separate Configuration Management Plan 
Respondent 9: 

a. Not necessary for ACAT IV, contractors are capable of doing this without being told to 
Respondent 14: 

a. Not sure what it is 
Respondent 16: 

a. Never prepared any separate CM Plan for my program 
Respondent 22: 

a. A lot of work to prepare and then not follow 
Respondent 48: 

a. Complexity and parts count of system 
Respondent 49: 

a. Configuration Management is usually controlled on a business as usual basis, plans are not 
usually generated 

b. If a plan was generated there would be a cost savings 

c. If a plan was generated the schedule would be shortened 
Respondent 60: 

a. No, Configuration Management Plan must be specific to parties involved 
Respondent 64: 
a. ? 

Respondent 69: 

a. No, Configuration Management Plan was not required for bid sample testing. Configuration 
Management Plan was required from successful contractor as a CDRL requirement 

22. Don’t generate Independent Life Cycle Cost Estimate 
Respondent 3: 

a. No, this document doesn’t take long to prepare and is very useful 
Respondent 9: 
a. ? 

Respondent 14: 

a. In the R&D cycle, cost of the system is a guesstimate and seems useless to me. All the 
numbers that come out of it are just even further guesses 
Respondent 16: 

a. Always prepare this LCCE because it is an annex to the IPS and required for all milestones. 
Also, since we do not normally do COEA this is the major document to analyze the cost of the 
system 
Respondent 22: 
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a. No, program should have schedule and program costs, how about using that 
Respondent 48: 

a. Relative cost of product, degree of user support 
Respondent 49: 

a. No, LCCE are needed for budget purposes and COEAs 
Respondent 60: 

a. No, summarized program 
Respondent 64: 
a. ? 

Respondent 69: 

a. Concept for fielding this NDI item was to use interim contractor support while acquiring 
provisioning information and technical data as CDRL items under the production contract. 
After interim contractor support period an economic analysis will determine if transition to full 
organic support is in the Army’s best interest 
Respondent 70: 

a. No, I’m not confident that ILLCEs are always effective or accurate (due to strict format 
requirements), however realize their need in addressing cradle-to-grave issues. I think this area 
should be more flexible in preparation to ensure effective. Maybe more of an assessment and 
less concrete dollar figures, i.e., what are cost drivers over life cycle - discuss not estimate 

23. Obtain Milestone Decision Authority approval to solicit a contract prior to Milestone Decision 
Review 

Respondent 3: 

a. This would definitely accelerate program when one is waiting for an IPR meeting before 
awarding next phase of contract 
Respondent 9: 

a. Gets the item to the field a lot quicker 
Respondent 11: 

a. Yes, if favorable milestone decision is fairly positive 

b. Funding saved acting concurrently 

c. Time saved acting concurrently 
Respondent 14: 

a. No, risky, a milestone will be done upon completion of testing, doesn’t give you the ability 
to incorporate changes in contract unless by modification. Many contingencies and reduction 
can come out of an IPR, in certain cases maybe a good idea, to meet a schedule 
Respondent 16: 

a. Have never done this but can see the potential savings for time if the program is constrained 
by time 

Respondent 17: . ., f t j 

a. Contract award would be subject to. Would save time, especially on items with long lead 
times 

Respondent 21: 

a. No, too risky 
Respondent 22: 

a. Soliciting okay but award should wait until decision review 
Respondent 48: 

a. End of fiscal year award with late fiscal year MDR 
Respondent 49: 

a. No, changes at the milestone decision review could have contract implications 
Respondent 60: 

a. No, MDA approval not required to solicit for contract 
Respondent 64: 
a. ? 

Respondent 69: 

a. No, Milestone Decision Authority had test reports and independent assessments from all 
contractors bid samples tested before recommending production contract award and Type 
Classification Standard 

24. Utilize a single contract for Dem/Val and EMD 
Respondent 3: 

a. This is a must in order to have any type of acceleration 
Respondent 7: 

a. Lower risks - corporate history/knowledge of contractor helps reduce risks (and schedule) in 
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second iteration 
Respondent 9: 

a. We do this now, basically, it works for us 

b. Who has time (18+ months) to wait for procurement to award another contract 

c. Money sits unused while procurement and legal diddle with the contract 
Respondent 11: 

a. Yes, but I would have to have a huge level of confidence in the system and that the 
Milestone II decision would be favorable. If I was going to hold a Milestone I/III, no problem 

b. Funding saved to generate the new contract 

c. Time saved to generate the new contract 
Respondent 14: 

a. Saves time of preparing, soliciting, and awarding contract. If new contractor, there will be a 
learning curve, starting a new contract will probably cost more than continuing with the 
previous one 
Respondent 16: 

a. Normally done because of our NDI approach and the combining of Milestones I/III 
Respondent 17: 

a. Saves time by not re-soliciting, better to have same contractor - no learning curve in EMD 
phase 
Respondent 21: 

a. Eliminates preparation of follow-up contract, saves time/money, better learning curve effect 
Respondent 48: 

a. Technical risk well understood and controllable, availability of funds 
Respondent 49: 

a. This approach would allow you to develop the final configuration during Dem/Val and 
smoothly transition to EMD 

b. Cost for one contract award can be eliminated 

c. Schedule would be shortened because of one less contract award 
Respondent 60: 

a. Phases combined 
Respondent 64: 

a. Save cost, save time 
Respondent 67: 

a. We are utilizing this 
Respondent 69: 

a. No, contractors provided their systems for bid sample testing and supported them through 
the test at no cost to the Government. The production contract award was sufficient incentive 
for their participation 
Respondent 70: 

a. Regarding ACAT IV programs, the time to complete Dem/Val may be the time to award a 
contract. Purpose of separating is to bring a complex effort into manageable pieces thereby 
reducing risk. In ACAT IV we usually have much less complexity and are able to manage the 
Dem/Val and EMD with less risk and greater ease 

25. Utilize a single contract for EMD and First Production 
Respondent 3: 

a. It would be great to have a single contract to include Dem/Val, EMD, and Production 

b. It would reduce time and add consistency 

c. This would be a great incentive to the contractor as well 
Respondent 7: 

a. Have not done - but expect this to become the standard operating procedure in the future 
because of the new rules against using military specs and technical data packages 
Respondent 9: 

a. Great idea, lets me fix engineering problems before they hit high volume production 
Respondent 11: 

a. Yes, provided the first production was a contract option 

b. Save the funding associated with a new solicitation 

c. Save time to prepare the solicitation 
Respondent 14: 

a. Wastes time 

b. Costs for start-up with each new contract 
Respondent 16: 

a. Have never done 
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Respondent 17: . 

a. Expertise gained by contractor during EMD phase would be valuable in production, could 
use a less-than-100% technical data package. Shorter lead time to production, tooling and 
ordering materials could be done sooner 
Respondent 21: 

a. Eliminates follow-up contractor, can hold contractor responsible for meeting design-to-cost 
goals, assess production readiness better 
Respondent 48: 

a. Technical risk low and well understood 

b. Strong user support and need 

c. Funds available 

Respondent 49: , . .. ,,, 

a. With a low risk program that will not see many changes during EMD, this would be 
appropriate 

b. Cost for one contract award can be eliminated 

c. Schedule would be shortened because of one less contract award 
Respondent 60: 

a. No, cost factors not fully known until after EMD build 
Respondent 63: .... , 

a. No, development and production should be separate entities - production contract may be 
competed to reduce item cost 
Respondent 64: 

a. No, should foster competition in production contract 
Respondent 69: 

a. No, went from bid sample testing directly to a production contract award 
Respondent 70: 

a. No, I do not necessarily agree with concept of having first production by contractor 
developing system for small programs. 1 think TDP should be proven in production by another 
source to work out growing pains. Many problems in TDP may not come out if given to same 
contractor - just put off to later date. Idea ok if awarding two production contracts to two 
contractors 


26. Conduct production First Article Tests to validate development deficiency corrections 

Respondent 9: , 

a. We do this now, good opportunity for everyone to get many problems out m the open 

Respondent 11: 

a. I believe the validation of deficiency corrections should be done before the First Article 
Test, although if the relationship with the contractor is strong it may be worthwhile 

b. Could save test costs 

c. Could save up to 6 months 
Respondent 14: 

a. Objective of First Article Tests 

Respondent 16: , ____ w , . . . 

a. This is done with the recommendations made by the evaluators. TECOM has the materiel 
release authority so they normally check the FAT to see if the corrections have been made 
before they will do the materiel release 
Respondent 17: 

a. Allows you to skip one of the development phases. This was done on SICPS Rigid Wall 
Shelter, good way to prove-out minor corrections to problems in DT 
Respondent 21: 

a. Cost savings, schedule savings 
Respondent 22: 

a. Want to check production ASAP 
Respondent 48: 

a. Reduce scope and timeline requirements of Government DT/OT testing 
Respondent 49: . 

a.First Article Tests are usually conducted on a business as usual basis 
Respondent 60: 

a. First Article Test validates final product 


Respondent 63: . 

a. No, development and production should be separate entities - production contract may be 
competed to reduce item cost 
Respondent 64: 
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a. No, should foster competition in production contract 
Respondent 69: 

a. Conducting weapon firing tests (Government test) as part of Initial Production/First Article 
Tests to farther demonstrate system reliability which was not adequately demonstrated in bid 
sample testing 
Respondent 70: 

a. No, This is commonly done and unfortunately, with less than desirable results. Leads to 
band-aid fixes that may never be truly fixed and a dissatisfied customer 

27. Validate technical manuals/support system during first production instead of during EMD 
Respondent 3: 

a. Accelerates Type Classification 
Respondent 9: 

a. Good because engineering changes cause manual changes which cost money 

b. You don’t waste time on the manuals during EMD 
Respondent 11: 

a. No, I believe it would be risky to evaluate the manuals after EMD especially in todays 
environment where contractors manuals will be used as the military manuals 
Respondent 13: 

a. No, this would probably take too much time during production 
Respondent 14: 

a. Has to be done prior to IOT&E 
Respondent 16: 

a. ATCOM is responsible for our Technical Manuals and they are done in the production 
contract. Most of our designs are built for the first time in the production contract (new 
contractor, performance spec, etc.) 

Respondent 17: 

a. Usually, there is not enough RDTE funding to buy manuals during EMD. If you make 
design changes during First Production, the manuals will have to change anyway 
Respondent 21: 

a. If EMD and first production are performed under same contract or by same contractor 
Respondent 22: 

a. Better test of how support system will work 
Respondent 48: 

a. Reduce EMD timeline by deferring some if not all logistics activities to production 

b. Risk of deferment 
Respondent 49: 

a.This will ensure the technical manuals are applicable for the production units 
Respondent 60: 

a. Easy to do in EMD 
Respondent 63: 

a. The TDP is firmer/more mature during first production and avoids unnecessary change if 
conducted during EMD 
Respondent 64: 

a. Cost of first production items less than during EMD 
Respondent 69: 

a. Only preliminary operators manuals were required for bid sample testing. Operator through 
DS/GS manuals are being obtained as CDRL requirements under the production contract 

Other: 

Respondent 10: Use capstone Program Management Documentation such as TEMPS, ASRs, and ORDs 
a. Allows development or several related efforts without waiting for separate staffing and 
creation of multiple documents 

Respondent 10: Use system integration of existing DoD and commercial items vs. new development 
a. More bang for the buck, quicker time to field, meets the intent of getting equipment into 
soldiers’ hands 

Respondent 22: One big checklist of design requirements 

a. Too many separate regulations with too much room for interpretation, should be able to go 
down list and address each requirement 

Respondent 69: Cost Savings 
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a. Production unit costs were considerably under the independent Government cost estimate. 
The incentive of a multi-year contract and the inherent competition from the bid sample testing 
resulted in savings of approximately 40% of the IGCE 
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